• 0 Posts
  • 59 Comments
Joined 11 months ago
cake
Cake day: August 4th, 2023

help-circle



  • More than a decade ago a user came into #ubuntu-server on Freenode (now libera.chat ) and said that they had accidentally run “rm -rf /* something*” in a root shell.

    Note the errant space that made that a fatal mistake. I don’t remember how far it actually got in deleting files, but all of /bin/ /sbin/ and /usr/ were gone.

    He had 1 active ssh connection, and couldn’t start another one.

    It was a server that was “in production”, was thousands of miles away from him, and which had no possibility for IPMI / remote hands.

    Everyone (but me) in the channel said that he was just SoL and should just give up.

    I stayed up most of the night helping him. I like challenges and I like helping people.

    This was in the sysv-init (maybe upstart) days, and so a decent number of shell scripts were running, and using basic *nix commands.

    We recovered the bash binary by running something along the lines of

    bash_binary_contents="$( </proc/self/exe)"
    printf "%s" > /tmp/bash
    

    (If you can access “lsof” then “sudo lsof | grep deleted” will show you any files that are open, but also “deleted”. You may be surprised at how many there are!)

    But bash needed too many shared libraries to make that practical.

    Somehow we were able to recover curl and chmod, after which I had him download busybox-static. From there we downloaded an Ubuntu LiveCD iso, loop mounted it, loop mounted the squashfs image inside the iso, and copied all of /bin/ , /sbin/ , /etc , and so on from there onto his root FS.

    Then we re-installed missing packages, fixed up /etc/ (a lot of important daemons, including the one that was production critical, kept their configuration files open, and so we were able to use lsof to find the magic symlinks to them in /proc/$pid/fd/ and just cp them back into /etc/.

    We were able to restart openssh-server, log in again, and I don’t remember if we were brave enough to test rebooting.

    But we fucking did it!

    I am certainly getting a lot of details wrong from memory. It’s all somewhere at irclogs.ubuntu.com though. My nick was / is Jordan_U.

    I tried to find it once, and failed.


  • It’s at least gotten a bit better.

    There was a time when Photoshop and other programs used a copy-protection scheme that overwrote parts of grub, causing the user not to be able to boot Linux or Windows.

    They knew about it, and just DGAF. I don’t remember their exact FAQ response, but it was something along the lines of “Photoshop is incompatible with GRUB. Don’t dual boot if you use Photoshop.”

    Grub still has code for BIOS based installs that uses reed-solomon error correction at boot time to allow grub to continue to function even if parts of its core.img were clobbered by shitty copy protection schemes for Windows software.








  • Interstellar_1@pawb.social

    Sorry again. I wrote this last comment (and this one, TBH) from my phone and “–iso=s” should have been “–iso-8601=s” . I’ve edited my comment and the command should now work (Making a backup of your grub.cfg containing the date, to the second, in the filename. I did that to hopefully avoid you running the same command again after trying some fixes and accidentally clobbering your backup).


  • Ahh, sorry.

    For Fedora it looks like the default /etc/default/grub looks like this:

    GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="rhgb quiet" GRUB_DISABLE_RECOVERY="true" GRUB_ENABLE_BLSCFG=true

    ( Taken from https://discussion.fedoraproject.org/t/how-to-regenerate-etc-default-grub/72677/9 )

    If you’re using LVM / LUKS you may need additional kernel parameters, like resume=… for suspend to disk to work properly.

    Please, before doing anything else, post the output of the following:

    cat /proc/cmdline

    And make a backup of your existing grub.cfg with:

    sudo cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg-backup-$(date --iso-8601=s)

    Also, be sure that you have a LiveUSB on hand. You don’t want to be SOL if we break something and can’t boot again without fixing it first.


  • Jordan_U@lemmy.mltoLinux@lemmy.mlI can't edit /etc/default/grub
    link
    fedilink
    arrow-up
    2
    arrow-down
    2
    ·
    edit-2
    4 months ago

    What version of Ubuntu are you using?

    What is the output of the following command?:

    dpkg -l | grep grub

    If you urgently want your grub menu to default to the first entry that can be done first, but unless that’s needed I’d prefer to get to the root of the problem(s) and get a proper fix.


  • This should get you back to defaults:

    sudo cp /usr/share/grub/default/grub /etc/default/grub && sudo update-grub

    At some point you definitely did accidentally write to /etc/default/grub when you meant to write to /boot/grub/grub.cfg.

    There’s no shame in that; Grub’s configuration process is very confusing and counter-intuitive.

    Everybody who has used Linux long enough has stories of breaking their systems in sillier ways, and this didn’t even really break your system 🙂.


  • Pulseaudio used features of sound cards (most prominently the hardware read pointer) that ALSA/dmix alone never used.

    ALSA/dmix could allow you to get the same power savings as pulseaudio if you set the hardware ring buffer size to, say, 2 seconds.

    And that would be fine of you were just playing some music, but if you were also chatting and wanting to get prompt notification sounds they would always be delayed between 0 and 2 seconds depending on where the hardware read pointer happened to be when the system tried to play a notification sound.

    ALSA/dmix could also allow you to set a tiny buffer size. Then your music would play, and your notification sounds would play promptly too. But if you were just playing music your CPU would never be able to go into the lower power sleep states because it would need to wake up every centisecond to service the tiny ring buffer.

    That would kill your battery life.

    Pulseaudio’s (terribly named) “glitch free” audio feature was the first solution for Linux that allowed you to get power savings and low-ish latency. Your mp3 player filled up the ring buffer once every two seconds, and if a notification came in pulseaudio would look at where the hardware read pointer was, take the contents of the buffer that was just about to be read, and mix the notification sound into it, writing the newly mixed sound data to the buffer just before the sound card read it.

    So, from the user’s perspective nothing interesting seemed to happen, but they get better battery life and things like notifications or game sounds work like they expect them to.

    ALSA drivers would commonly advertise support for accurately and precisely reporting the position of the hardware pointer, but since nothing actually used that info before, many drivers gave incorrect results, which would only cause problems when using pulseaudio.