Trisquel_9 (Etiona) experiences random freeze from which there is no recovery
Laptop T420 Lenovo No.1 has been running several nMap scans at once without
incident for several days in a row, but today it froze four times, after which
it could not be booted up in any one of three operating systems. One OS is TQ_9
on /dev/sdb1 with a data volume on /dev/sdb2. Another two OS's are on sda:
One TQ_8 and another TQ_9, which I haven't been using because the data I'm
working with is on sdb2. The two OS's on sda haven't been performing well by
comparison.
I shut down Laptop T420 Lenovo No.1, removed the caddy occupying the USB-connected
DVD bay, and set about installing it in Laptop T420 Lenovo No.2, which has 7.7GB
of RAM in contrast with No.1's 3.7GB of RAM. No.2 is also blessed with 1600x900
pixels in its LED display, much more than No.1's display has.
After several cycles of removal/update-grub, replacement/update-grub, I managed to
install the SATA caddy in No.2 and run three nMap scans under TQ_8, but that has
now frozen as well ... I'll deal with that as soon as I finish posting the present
issue, which is that No.1 will only run on a TQ0_8 live flash drive.
I think I must update the grub menu, but that will involve saving the edited grub
menu to a suitable place on either dev/sda or dev/sdb before restarting without
the live TQ installation flash drive in place.
Thanks,
George Langford
A search on "grub rescue" revealed several sets of instructions; the following is a good start:
https://help.ubuntu.com/community/Grub2/Troubleshooting#grub_rescue.3E-1 which is linked
from this forum.
After managing to achieve a running OS on the subject 'puter from the grub rescue screen, I found
that a reboot gets me back to the grub rescue screen, with the message:
No such device: [very long HDD identifier]
Not finding that HDD listed in fdisk, I realize that it's the identity of the HDD on the SATA
caddy pulled from the DVD drive bay, and now replaced with the other T420's DVD drive bay.
After an Internet search on [grub "no such device"] I came upon this link with the specifics:
https://askubuntu.com/questions/143667/boot-error-no-such-device-grub-rescue
which led me through the steps to install a new instance of grub ... which worked OK. What may
have helped a lot is that "fdisk -l" informed me that my /dev/sda3 is the boot device, so I made
sure to use that location at which to apply askubuntu.com's instructions.
Getting back to the freeze problem, the system dutifully froze this morning.
I was actually looking at a Trisquel.org blog entry; the freeze initiated when
I triled to exit the session.
I'm attaching snippets of various logs that may help us elucidate its cause.
Trisquel_9 is the operating system.
The 'puter is a ThinkPad T420 with 7.t GB RAM and 1600x900 display resolution.
The boot process was started at 08:25:35 on June 21, 2021.
I had to stop it by holding the power button down at 08:25:49.
| Attachment | Size |
|---|---|
| seat0-greeter.log-06212021.txt | 1.67 KB |
| Auth.log-06212021.txt | 8.38 KB |
| Var-Syslog-Freeze-06212021.txt | 32.81 KB |
I see nothing wrong in those logs. Random freezes are often due to faulty RAM. Have you tested your RAM? You can install "memtests86+" (in Trisquel's repository) and let it test you RAM during one full night. Memtest86+ is to be launched from the menu of GRUB, the bootloader, that lists the installed operating systems right after the computer switches on. You may have to press [Esc] to have GRUB's menu appear. If memtest86+'s blue screen turns red, your RAM is defective. Instead of installing memtest86+, you can test your RAM with any live ISO that includes memtest86+.
Memtest freezes in the first second of scanning, either of the first two choices
and even with F1 (fail safe). I'll look into the configuration.
Does Memtest86+ itself freezes or do you simply mean the screen turns red to indicate the RAM is defective? Either way, you certainly have a problem with your RAM (or with your motherboard?). Indeed, Memtest86+ is completely independent from Trisquel. It is a system dedicated to testing your RAM.
If you have several RAM sticks, unplug them all but one (a different one every time) to identify what is the faulty stick. You can then replace it, or work with less RAM.
Memtest86+ just freezes without any hysterics with both sticks in place. Swapping them in their respective
slots gives the same result in Memtset86+.
With just one of each, Memtest86+ runs OK; I finished one test last night. Tonight it'll be the other one.
After some time elapsed, I downloaded an earlier version of Memtest86 (v4.3.7) because 5.01 disappeared from
a second Lenovo T420, into which I added the noncompliant 4GB RAM stick to complement the existing 4GB RAM
stick. It's now working on that T420 with both 4GB sticks installed. The older Memtest is now on a bootable
USB stick.
On the second Lenovo T420 that older Memtest on its USB stick kinda works, but it cannot be stopped with
the ESC key, nor by typing ESC, and it doesn't identify the RAM sticks or their characteristics.
Therfore I've set about installing the Memtest86+ that is clearly present on one of the Trisquel_9 boot
sequence at /bootand and also in /etc/grub.d as 20_memtest86+, but that last file is beyond my pay grade.
Typing sudo apt-get install memtest86+ reveals that it's already installed, but not to the
extent that running update-grub brings it into the boot menu.
Haven't progressed that far ... but the Live Trisquel_9 USB stick is a viable alternative; I'll try it tonight.
Thank you for your attention to my plight.
The two T420's which I subjected to Memtest86+ 5.01 and Memtest 4.3.7 have error-free RAM.
The first is using the second RAM stick of the original incompatible pair.
The second T420 now has the first RAM stick plus its original 4GB RAM stick. All sticks are 4GB.
I'll next swap both 4GB RAM sticks from the T420 where they work OK to the other T420
where the first two RAM sticks didn't get along. Memtest will either run OK or not.
The original T420 that was the subject of this inquiry apparently has a faulty motherboard,
because Memtest86+ freezes within the first second with two identical 4GB RAM sticks.
Testing the second T420 with those two Samsung 4GB, 1333Mhz RAM sticks is on hold until I can
somehow persuade grub to include Memtest86+ in its startup menu.
To make a long story short, here's what works to shoehorn Memtest86+ into the grub menu:
Grub must be updated from the topmost (i.e. boot) operating system in the grub
menu, and Memtest86+ must be installed in that operating system.
When I transferred two, 2GB RAM sticks with equal specifications into the T420 with the "faulty"
motherboard, Memtest86+ is now evaluating that memory without freezing. The CPU temperature
reported by Memtest86+ for this T420 is about 20 degrees C more than for the second T420 and
its two, 4GB RAM sticks. Go figure.
I'm writing this post from a third T420 from which I retrieved those two, 2GB RAM sticks. It's
now a candidate for installation of a pair of new 4GB RAM sticks.
After running Memtest86+ on all three T420's with various combinations of RAM sticks, including
a new pair delivered in two hours from a famous oligarch's superstore, it turns out that two of
the three 'puters have error-free memories, but the third cannot tolerate pairs of RAM sticks,
even when they have identical specifications. It's relegated to the parts department until I
find a way of troubleshooting its motherboard.
That said, Trisquel_9 continues randomly to freeze. Yesterday it stopped a set of nine scripts
processing about 200,000 IPv4 addresses, which scripts I'm attempting to run on a daily basis
to evaluate the frequency with which some of those addresses are resolving to different PTR's
and/or their initially resolved PTR's turn out to have different addresses upon re-resolving
a short time later.
Where in yesterday's logs should I find what was going on when that freeze occurred ?
Another linux secret handshake comes to light, at least to this neophyte:
https://en.wikipedia.org/wiki/Magic_SysRq_key
Another linux secret handshake comes to light, at least to this neophyte:
https://en.wikipedia.org/wiki/Magic_SysRq_key
Returning to syslog.1; I had started the scripts around 12:14; the freeze happened at 13:17:
Jul 4 12:13:06 george-ThinkPad-T420 systemd-resolved[458]: message repeated 372 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.] <== NXDOMAIN responses are normal with nmap - amenex
Jul 4 12:17:01 george-ThinkPad-T420 CRON[2428]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 4 12:30:38 george-ThinkPad-T420 systemd[1]: Reloading.
Jul 4 12:30:39 george-ThinkPad-T420 kernel: [ 2685.657570] JFS: nTxBlock = 8192, nTxLock = 65536
Jul 4 12:30:39 george-ThinkPad-T420 kernel: [ 2685.800129] SGI XFS with ACLs, security attributes, realtime, no debug enabled
Jul 4 12:30:40 george-ThinkPad-T420 kernel: [ 2686.357995] sdb: sdb1 sdb2 sdb3
Jul 4 12:31:35 george-ThinkPad-T420 wpa_supplicant[754]: wlx00c0ca82ec36: WPA: Group rekeying completed with 00:7f:28:26:11:a1 [GTK=CCMP]
Jul 4 12:31:39 george-ThinkPad-T420 systemd[1]: Reloading.
Jul 4 13:03:33 george-ThinkPad-T420 wpa_supplicant[754]: wlx00c0ca82ec36: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-49 noise=9999 txrate=65000
Jul 4 13:03:59 george-ThinkPad-T420 systemd[1]: Started Run anacron jobs.
Jul 4 13:03:59 george-ThinkPad-T420 anacron[3499]: Anacron 2.3 started on 2021-07-04
Jul 4 13:03:59 george-ThinkPad-T420 anacron[3499]: Normal exit (0 jobs run)
Jul 4 13:10:22 george-ThinkPad-T420 kernel: [ 5068.242157] perf: interrupt took too long (2557 > 2500), lowering kernel.perf_event_max_sample_rate to 78000
Jul 4 13:17:01 george-ThinkPad-T420 CRON[3773]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
kernlog.1 gave us this for the applicable time period:
Jul 4 12:30:39 george-ThinkPad-T420 kernel: [ 2685.657570] JFS: nTxBlock = 8192, nTxLock = 65536
Jul 4 12:30:39 george-ThinkPad-T420 kernel: [ 2685.800129] SGI XFS with ACLs, security attributes, realtime, no debug enabled
Jul 4 12:30:40 george-ThinkPad-T420 kernel: [ 2686.357995] sdb: sdb1 sdb2 sdb3
Jul 4 13:10:22 george-ThinkPad-T420 kernel: [ 5068.242157] perf: interrupt took too long (2557 > 2500), lowering kernel.perf_event_max_sample_rate to 78000
auth.log.1 started with a bunch of nmap initializations on nine scripts leading up to 12:14, but I hadn't yet extended the
password timeout with visudo, so there was another flurry of paswsword actions on my part immediately before the freeze:
Jul 4 12:13:54 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:14:02 george-ThinkPad-T420 sudo: george : TTY=pts/9 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL IPv4s.2004-2013.i.txt
Jul 4 12:14:02 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:17:01 george-ThinkPad-T420 CRON[2427]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 4 12:17:01 george-ThinkPad-T420 CRON[2427]: pam_unix(cron:session): session closed for user root
Jul 4 12:30:38 george-ThinkPad-T420 polkitd(authority=local): Operator of unix-session:c2 successfully authenticated as unix-user:george to gain ONE-SHOT authorization for action org.gnome.gparted for unix-process:2498:267435 [/bin/sh /usr/sbin/gparted] (owned by unix-user:george)
Jul 4 12:30:38 george-ThinkPad-T420 pkexec: pam_unix(polkit-1:session): session opened for user root by (uid=1000)
Jul 4 12:30:38 george-ThinkPad-T420 pkexec[2505]: george: Executing command [USER=root] [TTY=unknown] [CWD=/home/george] [COMMAND=/usr/sbin/gparted]
Jul 4 12:30:53 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:30:53 george-ThinkPad-T420 sudo: george : TTY=pts/8 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:30:53 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:32:03 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:32:03 george-ThinkPad-T420 sudo: george : TTY=pts/5 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:32:03 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:32:54 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:32:54 george-ThinkPad-T420 sudo: george : TTY=pts/4 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:32:54 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:32:59 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:32:59 george-ThinkPad-T420 sudo: george : TTY=pts/9 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:32:59 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:33:33 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:33:33 george-ThinkPad-T420 sudo: george : TTY=pts/6 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:33:33 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:33:40 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:33:40 george-ThinkPad-T420 sudo: george : TTY=pts/7 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:33:40 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:34:53 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:34:53 george-ThinkPad-T420 sudo: george : TTY=pts/3 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:34:53 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:38:29 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:38:29 george-ThinkPad-T420 sudo: george : TTY=pts/1 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:38:29 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 12:39:35 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 12:39:35 george-ThinkPad-T420 sudo: george : TTY=pts/2 ; PWD=/media/george/b8a1ee74-96d2-407a-8d4e-98df997fcbe3/george/Thumb256E/Documents/GeorgesBasement.com/SpamCop-QuickReports/QR-07042021 ; USER=root ; COMMAND=/usr/bin/nmap -Pn -sn -T2 --max-retries 16 -iL -
Jul 4 12:39:35 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 4 13:08:14 george-ThinkPad-T420 sudo: pam_unix(sudo:session): session closed for user root
Jul 4 13:17:01 george-ThinkPad-T420 CRON[3772]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 4 13:17:01 george-ThinkPad-T420 CRON[3772]: pam_unix(cron:session): session closed for user root
Looks bad for those cron jobs starting at the :17 minute mark every hour ... another freeze-like event
happened when I tried to post this reply a little while ago. I was able to save the Preview text to HDD.
Some more poking around reveals:
george@george-ThinkPad-T420:~$ sudo grep CRON /var/log/syslog
Jul 5 14:31:21 george-ThinkPad-T420 cron[789]: (CRON) INFO (pidfile fd = 3)
Jul 5 14:31:21 george-ThinkPad-T420 cron[789]: (CRON) INFO (Running @reboot jobs)
Jul 5 15:17:01 george-ThinkPad-T420 CRON[2157]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 5 15:29:03 george-ThinkPad-T420 cron[775]: (CRON) INFO (pidfile fd = 3)
Jul 5 15:29:03 george-ThinkPad-T420 cron[775]: (CRON) INFO (Running @reboot jobs)
This link suggests that the implicated cron job is otherwise normal:
https://askubuntu.com/questions/635704/strange-root-cronjob-ive-never-set-it
But I couldn't make enough sense of the options for "ls" to proceed past this:
ls -hacClNRtZ --author --time-style=full-iso /etc/cron.hourly
/etc/cron.hourly:
total 20K
drwxr-xr-x 149 root root root ? 12K 2021-07-04 12:06:54.107574725 -0400 ..
drwxr-xr-x 2 root root root ? 4.0K 2021-06-24 12:51:01.665066801 -0400 .
-rw-r--r-- 1 root root root ? 102 2021-06-24 12:49:14.361072098 -0400 .placeholder
The data in the second column of each line don't ring any bells ...
Magic Banana, posting too close to my recent edit, wonders what's written in /etc/cron/hourly,
but my anticipate response doen't really uncover very much, though there appear to be 20k of
either bytes or entries ...
On the good news front, I looked at the syslog entry for a successful run of the same nine
scripts, albeit in Trisquel_8, and the same cron job is getting run there also, faithfully
on the :17 minute of every hour, and there are no complaints expressed in the log over the
period from 10:38 tp 14:58 today.
Here's the response to Magic Banana's question for today's data:
ls -hacClNRtZ --author --time-style=full-iso /etc/cron.hourly
/etc/cron.hourly:
total 20K
drwxr-xr-x 155 root root root ? 12K 2021-07-05 09:01:06.495393062 -0400 ..
drwxr-xr-x 2 root root root ? 4.0K 2020-12-04 15:01:32.951236198 -0500 .
-rw-r--r-- 1 root root root ? 102 2020-12-04 14:58:48.674338162 -0500 .placeholder
This response is from the Flidas log files; the prior reponse was from the Etiona log files.
Does the Flidas edition of Trisquel use different hardware than the Etiona edition ?
Flidas doesn't freeze on the same T420 as Etiona does while running the same nine scripts.
Intel has a diagnostic tool for their processors:
https://downloadcenter.intel.com/download/19792
but there's no listing for a linux operating system, though my T420's processor is covered.
Edit: Flidas did freeze ... twice, while running the aforementioned nine scripts. Consequently,
I moved the machinery over to the other T420 that tolerates two, 4GB RAM sticks, hot-plugged
the USB-3.0-connected SATA HDD, ran update-grub to obtain a fully populated GRUB menu, and ran
the nine nmap scripts on the Etiona OS on that HDD without incident. I didn't have a chance to
attempt running the same scripts on any installed Flidas because none of them will boot up on
this T420. Tomorrow's run will tell.
After more than three weeks of freeze-less operation, Etiona simply stopped this morning,
leaving no trace that I could find in any of the /var/log files.
I had just completed a security update with Software Updater, followed by a reboot.
A couple of instances of each of Leafpad and another Mate Terminal were running, and I was in the
process of copying & pasting an IPv4 address from a diff file in Leafpad to a bash window in
which I was constructing a grep command to locate information relating to that address in
a large (250MB) text file, probably stored as a temp file in RAM. The same steps had already
completed without incident. The grep operations were taking almost no time or RAM to complete;
each one was using the "| more" limitation.
Anther freeze; again none of the logs in /var/logs have timestamps within 12 hours,
in spite of all my computer activities between 08:30 and 20:30 on 07/30/2021. At the
start of this thread syslog _did_ cover the time period of the freeze.
Following Magic Banana's advice to IBM1130: https://trisquel.info/en/forum/problem-after-latest-update#comment-159142 that pointed to another page https://trisquel.info/en/forum/problem-after-latest-update#comment-159109 I installed "linux-image-generic-hwe-18.04" and rebooted.
Afterwards, I noticed that /var/log/syslog again keeps up with current events.
However, one of those logged events was disconcerting:
Jul 31 09:39:41 george-ThinkPad-T420 gvfsd-metadata[1584]: g_udev_device_has_property: assertion 'G_UDEV_IS_DEVICE (device)' failed
Jul 31 09:39:49 george-ThinkPad-T420 gvfsd-metadata[1584]: message repeated 15 times: [ g_udev_device_has_property: assertion 'G_UDEV_IS_DEVICE (device)' failed]
An Internet search on g_udev_device_has_property: assertion 'G_UDEV_IS_DEVICE (device)' brings up
a two-year-old Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832138 which is reminiscent of some recent Trisquel issues ...