Posts | sgala.com . website

IPv6 come abbiamo fatto in Ehiweb - 6lottino 2022 - MIX-it

Wed, 05 Apr 2023 21:17:06 +0100

A Ottobre 2022 sono stato ospite del 6lottino di MIX-it (https://mix-it.net/) dove ho presentato come abbiamo adottato IPv6 sulla nostra rete d’accesso nazionale (ADSL/FTTC/FTTH) di Ehiweb(https://www.ehiweb.it) a livello massivo su tutta la clientela.

Ho descritto il nostro piano di adozione, l’approccio e alcune sfide che abbiamo dovuto affrontare e risolvere.

Note

TL;DR Link al pdf della presentazione

Fetch a service status detail via CheckMk REST API

Mon, 27 Mar 2023 21:17:06 +0100

I had to write a script that fetch from a CheckMk monitored service the status of that service and do something.

I tried to ask how to do this to several friends that are using CheckMk… no one had ever do this before ;(

In my real scenario I need to fetch a specific service status:

Critical
Contains some specific text matching in the perf data field.

TL;DR;

USERNAME=$USERNAME$
PASSWORD=$PASSWORD$
curl -s -G \
 --header "Accept: */*" \
 -H "Authorization: Bearer $USERNAME $PASSWORD" \
 --data-urlencode 'columns=host_name' \
 --data-urlencode 'columns=perf_data' \
 --data-urlencode 'query= \
 { "op": "and", "expr":[{"op": "=", "left": "host_name", "right": "$HOSTNAME$"}, \
 { "op": "=", "left": "description", "right": "$SERVICE_NAME$"}, \
 { "op": "=", "left": "state", "right": "2" }, \
 { "op": "~~", "left": "perf_data", "right": "$SOMETHING_TO_FILTER" } ]}' \
 "$CHECKMK_URL$/$SITE_NAME"/check_mk/api/1.0/domain-types/service/collections/all

In this example the shell lines do a CURL request to the CheckMk REST API and fetch only the status of the service $SERVICE_NAME$ that are in a Critical status and contains the word $SOMETHING_TO_FILTER$. The query it’s very useful as you can fine match what do you want without write any code outside. The query is wrote in the livestatus “language”. As you can see there are multiple columns added, these are not provided by default in the results, so I added them because were needed in my scenario.

Some links

a post I did on checkmk forum about some issues resolving this https://forum.checkmk.com/t/rest-api-fetch-service-status-missing-details-summary/37613
more example/docs https://kb.checkmk.com/display/KB/REST-API+Code+Examples

IPv6 RADIUS Accounting

Wed, 29 Jun 2022 12:48:06 +0100

I’m struggling to understand why on accounting packets of my Cisco ASR1001-X with a dual stack configuration, the Delegated-IPv6-Prefix is not present at all.

In my configuration I’m using DHCPv6 PD and Delegated-IPv6-Prefix sent from the RADIUS server in Access-Accept response in every request.

I found that interesting article of and found the reason about it.

Basically you have to enable the feature with this command, keep in mind that it’s only supported from IOS 15.1.

aaa accounting include auth-profile delegated-ipv6-prefix

Here a screenshot of the Cisco documentation and to the official documentation about that command.

IPv6 subnetting in command line with sipcalc

Mon, 30 May 2022 09:17:06 +0100

I had to put my hands on my IPv6 network after lots of years… my memories are not so good…how to subnet a IPv6 network in command line?

What to do

I need to generate a pool of network /64, /60, /56 and /48 to assign to new customers from larger CIDR. Quickly and without painass.

Sipcalc

it’s an amazing tool for IPv4 and IPv6 calculations.

From the I found a perfect option to do this:

* -S, --v6split=MASK (IPv6)
 Split the current network into subnets of MASK size. MASK must be given in
 CIDR form, either with or with the '/' character.

Problem

For example, I have a /48 network and I want to split into /56

> sipcalc -u 2a01:b600:4000::/48 -S 56 |head -n 50

-[ipv6 : 2a01:b600:4000::/48] - 0

[Split network - verbose]
-[ipv6 : 2a01:b600:4000::/48] - 0

[IPV6 INFO]
Expanded Address - 2a01:b600:4000:0000:0000:0000:0000:0000
Compressed address - 2a01:b600:4000::
Subnet prefix (masked) - 2a01:b600:4000:0:0:0:0:0/56
Address ID (masked) - 0:0:0:0:0:0:0:0/56
Prefix address - ffff:ffff:ffff:ff00:0:0:0:0
Prefix length - 56
Address type - Aggregatable Global Unicast Addresses
Network range - 2a01:b600:4000:0000:0000:0000:0000:0000 -
 2a01:b600:4000:00ff:ffff:ffff:ffff:ffff

-
-[ipv6 : 2a01:b600:4000::/48] - 0

[IPV6 INFO]
Expanded Address - 2a01:b600:4000:0100:0000:0000:0000:0000
Compressed address - 2a01:b600:4000:100::
Subnet prefix (masked) - 2a01:b600:4000:100:0:0:0:0/56
Address ID (masked) - 0:0:0:0:0:0:0:0/56
Prefix address - ffff:ffff:ffff:ff00:0:0:0:0
Prefix length - 56
Address type - Aggregatable Global Unicast Addresses
Network range - 2a01:b600:4000:0100:0000:0000:0000:0000 -
 2a01:b600:4000:01ff:ffff:ffff:ffff:ffff

-
-[ipv6 : 2a01:b600:4000::/48] - 0

[IPV6 INFO]
Expanded Address - 2a01:b600:4000:0200:0000:0000:0000:0000
Compressed address - 2a01:b600:4000:200::
Subnet prefix (masked) - 2a01:b600:4000:200:0:0:0:0/56
Address ID (masked) - 0:0:0:0:0:0:0:0/56
Prefix address - ffff:ffff:ffff:ff00:0:0:0:0
Prefix length - 56
Address type - Aggregatable Global Unicast Addresses
Network range - 2a01:b600:4000:0200:0000:0000:0000:0000 -
 2a01:b600:4000:02ff:ffff:ffff:ffff:ffff

As you can see the result are the /48 network splitted with lots of info that can be useful or not.

As i need only the IPv6 address generated with the CIDR this is a output more useful (in this example the first 10 nets):

> sipcalc -u 2a01:b600:4000::/48 -S 56|grep 'prefix'|head -n 10
Subnet prefix (masked) - 2a01:b600:4000:0:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:100:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:200:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:300:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:400:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:500:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:600:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:700:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:800:0:0:0:0/56
Subnet prefix (masked) - 2a01:b600:4000:900:0:0:0:0/56

It’s done. You can generate all type of network split you want modifying the network size and split size.

Thanks for reading!

Java process >100%cpu, find the guilty thread

Sun, 09 Jan 2022 09:17:06 +0100

It’s a rainy friday… and my checkmk monitoring alert me about this:

It’s a application server VM where are running several Python and Java softwares belonging to an our product. Seeems that something happened and now the CPU load is very high than normal. It’s not normal and seems not temporarly.

First dig

Found via top/htop that the responsible is as single Java process… It’s an our legacy and very consolidated software that is not modified ealier. We need to find what is causing this strange usage and find the root cause in order to mitigate. That Java process have hundreds of threads and is very complicated…

Telemetry

All the telemetry data of the application are under the normal limits. No abuses or peaks.

Logs / Debug

Normal logs give us no useful information. All is working fine. Enabled the debug mode: no useful info, we see normal traffic belonging correctly to the application and all is working fine.

So?

It’s time to trace the Java process… who ever do it? It’s the time for the first time…

How to trace a java process?

I really don’t know. I need to found the thread responsible of the abnormal CPU usage. Does exist a TOP like command for a Java process? Asking to Google I found this .

Is very simple:

jvmtop $PID

Executed on my system and the output is very similar to top/htop command. Unfortunately all top consuming CPU threads are named Receiver-179, Receiver-190, etc… The good news is that now we know the Thread name… the bad news is that these thread names does not belong to our codebase but are thread named by a third parties library…

We need more info…how?

Googling… found the tool already bundled in the JDK by default.

Jstacking my Java app

It’s time to jstacking my java app. This tool give your as output the full stack trace of the java process with a output like this:

"Thread-514" #538 prio=5 os_prio=0 tid=0x00007ffab06c5800 nid=0x5db0 runnable [0x00007ffa81758000]
 java.lang.Thread.State: RUNNABLE
 at java.lang.Thread.yield(Native Method)
 at org.smpp.util.ProcessingThread.stop(ProcessingThread.java:162)
 at org.smpp.Receiver.stop(Receiver.java:219)
 at it.besms.server.SMSCSession.run(SMSCSession.java:142)
 - locked <0x00000000ee6efcd0> (a it.besms.server.SMSCSession)
 at java.lang.Thread.run(Thread.java:748)

"Receiver-179" #532 prio=5 os_prio=0 tid=0x00007ffacc048800 nid=0x5dac runnable [0x00007ffa81354000]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 at java.net.SocketInputStream.read(SocketInputStream.java:171)
 at java.net.SocketInputStream.read(SocketInputStream.java:141)
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
 - locked <0x00000000ee71f1d8> (a java.io.BufferedInputStream)
 at org.smpp.TCPIPConnection.receive(TCPIPConnection.java:413)
 at org.smpp.ReceiverBase.receivePDUFromConnection(ReceiverBase.java:197)
 at org.smpp.Receiver.receiveAsync(Receiver.java:358)
 at org.smpp.ReceiverBase.process(ReceiverBase.java:96)
 at org.smpp.util.ProcessingThread.run(ProcessingThread.java:191)
 at java.lang.Thread.run(Thread.java:748)

With the information got from jvmtop, I know the thead names and with this output I can find the classes involved on these threads and debug more deeply on the source code.

These threads are belonging to a Java SMPP library. The classes involved are regarding the inbound SMPP traffic.

Root cause

Now we can point the attention to the right direction: Inbound SMPP traffic.

We checked more deeeply the logs of that specific part and found a single host that is trying to authenticate about 20 times/sec and it was refused by the application for a internal reason. Analyzing the source code we found the root cause of the issue and was a bug on the library that cause a abnormal usage of CPU of theese threads if they are closed too quickly.

Mitigation

In order to mitigate the issue we dropped the host traffic.

Afterthat we fixed the issue on the source code and deployed a final fix.

Debug/trace a Java process is not a so trivial issue! ;)

Thanks for reading!

NTP out of sync each night on ESXi

Tue, 14 Dec 2021 23:17:06 +0100

In the last weeks I installed CheckMk to try it. I already added to CheckMK monitoring several hosts, mainly VM with its Linux Agent.

Since I’m trying it, I’m not giving it so much attention, right now. Yesterday, some late alarms got my attention. Seems that some servers are with the clock drifted, also if they have NTP installed and (I hope)configured correctly.

It’s time to dig.

First

Pattern

During digging I found a strange pattern. While I was on ssh on a VM, another one had the same NTP alert. Very very strange. This is an alarm bell. Shouldn’t a specific VM issue but a quite large one. It’s time to use the powerful of CheckMK. I did a search for the service alert NTP and found a very interesting behavior on it. This is an example of the output

I can see:

every ~24h it happens
it has been resolved automatically after sometime
it happens in a lot of VM But why?!

Matching ideas

It cannot be a NTP issue on a specific VM as several VM have the same issue. It cannot be a Debian version specific issue or systemd issue as it happens also on Devuan and on several different VM with different Debian releases. So? Idea: Backups! The Veeam backup jobs run exactly on these hours… but does not make sense…on the basis of the knowledge that already I have… Googled and…found!

VMWare NTP sync during snapshots

I discovered that by design vMware when take snapshots (and Veeam take snapshots for doing its backup) it sync the time of the ESXI hosts to the VM via vMware Tools. This is correctly documented here, the interesting part is:

One-off time sync: Guest system clock is synchronized to host time upon specific VM life-cycle events that can cause guest > clock to become incorrect, such as resuming from vMotion or a snapshot. This capability is recommended for use and turned > on by default. VMware Tools does not set time backwards (when guest time is ahead of the host), except once when periodic time > synchronization is turned on.

Clock assessment on the VMWare infrastructure

Checked the config on each ESXi hosts:

NTP is configured on each ESXi;
the clock is wrong on all ESXi; …

Checking more deeply on each ESXi hosts, I found that the NTP servers configured on them are belonging to a my temporary infrastructure decommissioned. These NTP servers are definitively not working… and this is the reason why all clocks are wrong.

Final timeline of the events

What’s happening:

all ESXi servers have bad NTP servers
all ESXi clocks drifted
at every Veeam backup, VMWare synchronize the ESXi clock to the VM (with a wrong clock)
checkmk discover the drift and catch an alert
after some minutes NTPd servers running on each VM correct the clock of the VM
checkmk alert are cleared

This was happening cyclically each night, each, backup window… the mystery has been found.

I fixed the NTP servers with valid ones on each ESXi, the clock has been corrected and the issue has been resolved definitively.