My internship is coming to an end and I think my boss is testing my limits.

So, in the beginning of this week, he assigned me a non reproducible bug that has been causing trouble to the whole team for months.

Long story short, when we edit or create a planned order from the backend, once in fifteen, a product is added to the list and "steals" the quantity from another product.
Everyone in the company has experienced this bug several times but we never got to reproduce it consistently.

After spending the whole week analyzing the 9 lines of JS code handling this feature, reading tons of docs and several libraries source code. I finally found a fix by "bruteforce testing" with selenium and exporting screenshots, error logs and snapshots of the html source.

This has been intense but was worth the effort, first, I fix a really annoying bug and second, I learned a lot of things and improved my understanding of Javascript.

Fuck. I can't take this shit anymore.

There was a project where we had to implement third-party system for government agency processes management. For some reason, probably because my work is cheap for my boss, the task was assigned to me. Just as a reminder, I'm a .NET Dev. Zero experience in server management. Zero experience in external services implementation.

Anyway, system producent, also an government agency, got angry, becasue they can only earn money on implementation. They have to give the software to other agencies for free. Because of that I've got client program, incomplete documentation and broken scripts for database creation. It took me 2 months to get it all to work but at the end client was happy, my boss got paid and I've got 500 PLN (~130 USD) bonus.

Everything was fine for a while, but after a month server has started freezing everyday, some time before 7 am. The only way I found to make it work again was to restore snapshot made everyday at 10 pm. For a month I was waking up earlier and restored snapshot, and after that my boss took it upon himself. I tried few times to find a bug and fix it, but to no effect. Even person with much more experience with it tried to help but also couldn't find anything.

My solution? Copy all the data and configuration, create new machine, copy everything and check if the problem persists. If not, kill old server. Client won't even notice. But nooooooooo... It would cost my boss a bit of money and I'd need to work on it and he can't let it be, because I'm the only developer working on his flagship product. He'd rather wake up everyday and restore snapshot. Okay, as you wish.

And today, finally, everything went downhill. Snapshot wasn't created, server froze, backup can't be created. Nothing can be done. Client is furious, because they have had reported this problem and a few times restoration was too late and they couldn't work. No one knows how to fix it, I'm not working today (I'm still studying and am available only 2 days) and situation is really shitty.

BUT SURE. ITS BETTER TO RESTORE SERVER EVERYDAY THAN JUST FUCKING FIX IT.

Oh, also, there's no staging or any other real backup. We have snapshots for each day and that's that. Boss' order. Why do I even care...

I really want to.
I want to get away from Windows.
But I have yet to find a distro that works.

Today I tried again again.
I found out Linux Mint released version 19.
Snapshots integrated. Cool. I will try it.

Installed with Cinnamon. Looks nice. Everything is running fast.

Aaaaand I hate the mouse movement...
Why is there no 1:1 movement? Is acceleration on? Does not feel like it.
Ok. We can fix this right?
Opened the Mouse settings.
There is no way of deactivating mouse acceleration. Only customizing the amount of acceleration. What?

Ok customize it. No change at all.
Try extreme settings. Nothing.

Google for a solution. Says I should install dconf-editor and change settings there.

Install it, change the settings. Hey it works!
It is far from perfect but I can live with that.

Now the scroll wheel is so slow...
But there is no setting at all? Not even in dconf-editor.

Google the solution. Need to install imwheel and configure it. Really?
Okay will do. I wanna use this.

Finally. Mouse works as it should.

After all that, why is my lap so hot? Fucking hell the cpu seems to be burning.

Fuck that!
I am out! Back to Windows!

26 or so hours up now. And I've got a few stories to tell :) feel free to refresh your cup of coffee and take a seat.

Last few days I've been going into this odd place called intown.irl to get in touch with its inhabitants. An odd place I have to say. But in some cases quite rewarding, even got a MILF home with me and into bed at some point. Anyway...

3 days ago I think it is now? Thursday evening I took my laptop to this local bar where I had this issue about dihydrogen monoxide with one of the bartenders earlier (you'll find that rant on those keywords). Still wanted to visit it regardless though, as I met that first woman there earlier that approached me. Unfortunately I didn't see her there that day.

Some bald guy who was clearly drunk approached me. Many people were already giving curious looks at this laptop I brought to the bar. I finally tuned it up with the stickers from FOSDEM.. I'll put a picture of it in the comments. My theme was one of privacy (central), distributions and Google's open source initiative (which aligns with the keychain token I got from them as well). But of course.. that guy.. he thought that a pimped/riced laptop obviously meant that I was a hacker.

Guy went to the toilet.. went back.. and suddenly grabbed my laptop and turned it towards him. Boy was I never more smugly satisfied that those rubber pads on the bottom are quite resilient. Could've almost damaged my screen by trying to grab it like that. But it's a CCFL display.. so high voltage. If it were to become broken.. worth it. 😈

On it at the time was a terminal, pinging Google (had network issues at that bar, to the point where one of the - I think - staff members got up to me and offered the WiFi password and got to talk with me.. more on that later), and my usual Linux desktop along with the Arch anime wallpaper with the quote of Da Vinci.. simplicity is the ultimate sophistication. Of course the guy saw the terminal.. and probably reaffirmed.. yep, that's a hacker. At least he wasn't too wrong about the general term.. but the hat.. most likely he was wrong on that one.

Guy left with this question.. "you are a hacker, aren't you."
I replied to him: "No sir. I'm not a hacker. I've got no idea what you're talking about."
Guy kept looking at me weirdly for the whole night to come.

Back to that companion guy though. Mac user, yada yada.. but he told me about his backup solution. Apparently - I shit you not - he has not only the photos on his local device, he's also frequently backing them up in Time Machine (which I was really curious about whether it uses mirroring or snapshots.. he couldn't tell, lmk if you do) but not only that.. he was storing another offsite backup in that very bar, in case his house went on fire.

Now that is a proper backup scheme!!! If only more people were like that.

Seriously though.. that bald guy who took my laptop just like that... I just let it slide for that one time, but I tend to treat my machines as an extension of my very self. I think that was a very uncalled for move. Asshole...

How would you have reacted to such a thing? And.. maybe that's why we technologists don't get outside too often? Fucking everything is hacking these days if it's not Knopkes and Blinkenlights… Not every shell is a h4xx0ring console for h3kk1ng de fasbuk…

Got pulled out of bed at 6 am again this morning, our VMs were acting up again. Not booting, running extremely slow, high disk usage, etc.

This was the 6 time in as many weeks this happened. And always the marching orders were the same. Find the bug, smash the bug, get it working with the least effort. I've dumped hundreds of hours maintaining this broken shitheap of a system, putting off other duties to keep mission critical stations running.

The culprits? Scummy consultants, Windows 10 1709, and Citrix Studio.

Xen Server performed well enough, likely due to its open source origins and Centos architecture.

Whelp. DasSeahawks was good and pissed. Nothing like getting rousted out of bed after a few scant hours rest for patching the same broken system.

DasSeahawks lost his temper. Things went flying. Exorcists were dispatched and promptly eaten.

Enough. No consultants, no analysts, and no experts touched it. No phone calls, no manuals, not even a google search. Just a very pissed admin and his minion declaring blitzkrieg.

We made our game plan, moved the users out, smoked our cigs, chugged monster, and queued a gnu-metal playlist on spotify.

Then we took a wrecking ball to the whole setup. User docs were saved, all else was rm -r * && shred && summon -u Poseidon -beast Land_Cracken.

Started at 3pm and finished just after midnight. Rebuilt all the vms with RDP, murdered citrix studio (and their bullshit licenses), completely blocked Windows 10 updates after 1607, and load balanced the network.

So what do we get when all the experts are fired? Stabbed lightning. VMs boot in less than 10 seconds, apps open instantly, and server resources are half their previous usage state. My VMs are now the fastest stations in our complex, as they should be.

Next to do: install our mxgpu, script up snapshots and heartbeat, destroy Windows ads/telemetry, and setup PDQ. damn its good to be good!

What i learned --> never allow testing to go to production, consultants will fuck up your shit for a buck, and vendors are half as reliable over consultants. Windows works great without Microsoft, thin clients are overpriced, and getting pissed gets things done.

This my friends, is why admins are assholes.

At my previous job I used to deal directly with a client. This client wasn't particularly tech savvy although he was the head of IT department.

During the beta testing phase of the project. The client would regularly send me emails with snapshots of his laptop with application open accompanied by some amount of text to explain what the error/problem he was facing.

One fine day I got an email with a very wierd error. It was a module for attendance of employees based on which salaries were calculated. The mail had a snapshot and some text saying that he was unable to see the attendance for the dates 22, 23, 24 and so on...

I immediately replied with a snapshot of the calendar highlighting the date, which was the 21st of the month. Those attendance are empty because they haven't been marked yet.

No reply untill the next day with more "errors" 😂

You know the worst thing about being a freelancer? You're expected to wear every fucking hat and you don't get normal hours.

Over the past few days I have been working with a client of a client attampting to fix his server. He's running CentOS on VMWare and somehow ended up breaking the system.

Upon inspection there was no way to fix his system remotely. It wouldn't even boot in recovery mode. So we've been attempting to recover his data so that we can reinstall CentOS and not have to start completely from scratch.

So for the past 3 days straight I have been remotely logging in to a Debian Live CD and manually sending folders to a FTP server of his. He has somewhere close to 30 sites on this server, and upwards of 1 million files in total.

Yesterday either the system freaked out or he did something, but the entire fucking system stopped responding which forced me to reboot it, reinsert the live CD, reinstall evertything, and re-mount his broken systems drives.

Here we are 3 days in, we're still not done, and I'm getting slightly pissy because if you don't know Linux well enough to fix this shit yourself, you shouldn't be acting as your own sysadmin for 30+ sites.

Also, backups are a thing right? VMWare also has snapshots. I know the extra storage isn't cheap, but it's a hell of a lot cheaper than paying soemone like me $35/hr to go and fix all of your shitty mistakes.

Truncated the dev database today.

Thank god for snapshots. 😂😭

So recently I wanted to delete a corrupt jpg file.
I could not because "It's corrupted".
Oh dear windows...
...Go fuck yourself!

Oh and also recently I was out of space, so I had to free some by deleting the VMWare Snapshots that were taking up said space.
I couldn't...
...VMWare said it needs more space to be able to delete those files.
Fuck you too VMWare!

I deleted over a petabyte worth of snapshots from AWS today.

As a data hoarder this feels like genocide.

I'm the lead dev on this team. The project is split into multiple separate modules to comply with separation of concerns, and so new devs don't need the whole fucking codebase (risking them running away with everything) to contribute to the project as a whole.

So we don't need a fucking config file to enable and disable features.

So we don't need to upload a 500mb monolith every time we want to test a change.

So we can test old fucking versions of modules without merging it back into the entire codebase.

What did this fucking dev do? He was having one small issue with Maven. One. It wasn't updating his local snapshots to the correct Artifactory version.

He decided, instead of trying to fucking fix it: HEY, LETS IGNORE THE LEAD DEV'S DEMAND TO KEEP THEM SEPARATE. IM GOING TO MERGE THEM INTO ONE MODULE FOR SOME FUCKING REASON.

I refuse to continue working with this dev if he's going to sidestep my demands and undermine my authority. He wants to go it alone? Be my fucking guest. I'm not touching his shitty single-codebase monolithic monstrosity.

If this is going to be a regular fucking occurrence, he can eat a dick and choke on it.

man i wish i had brain cells.

i was trying to free some storage by deleting some btrfs snapshots, so i mounted the directory and started rm -rfing, as you do. a couple minutes in, i decide to reboot and make sure i didn't accidentally start deleting my root partition.

spoiler alert: i accidentally started deleting my root partition.

it only got up to deleting /data (where i keep my multimedia), but my whole /boot directory is gone. now I'm gonna have to spend a couple hours redoing stuff like fstab and grub to make things work again 😃

Ffs, HOW!?!? Fuck! I need to get this rotten bs out.

RDS at its max capabilities from the top shelf, works OK until you scale it down and back up again. Code is the same, data is the same, load is the same, even the kitchen sink is the same, ffs, EVERYTHING is the same! Except the aws-managed db is torn down and created anew. From the SAME snapshots! But the db decides to stop performing - io tpt is shit, concurrency goes through the roof.

Re-scale it a few more times and the performance gets back to normal.

And aws folks are no better. Girish comes - says we have to optimize our queries. Rajesh comes - we are hitting the iops limit. Ankur comes - you're out of cpu. Vinod thinks it's gotta be the application to blame.

Come on guys, you are a complete waste of time for a premium fucking support!

Not to mention that 2 enhanced monitoring graphs show anythung but the read throughput.

Ffs, Amazon, even my 12yo netbook is more predictable than your enterprise paas! And that support..... BS!

We're now down to troubleshooting aws perf issues rather than our client's....

!RANT

Oh, the SORROW that is JEST! 😡

Endless days have been swallowed by the abyss in my quest to configure Jest with TypeScript and ECMAScript modules instead of CommonJS. Triumph seemed within my grasp until - BAM! - suddenly the tool forgets what "import" or "export" means. And the kicker? On the CI, it still runs like nothing’s amiss!

Allow me to elucidate for the uninitiated: Jest is supposed to be a testing safeguard, a protective barrier insulating devs from the errors of their peers, ensuring a smooth, uninterrupted coding experience.

But OH, how the tables have turned when the very shield becomes the sword, stabbing me with countless, infuriating errors birthed from Jest’s own design decisions!

The audacity to reinvent the whole module loading process just to facilitate module mocking is mind-boggling! Imagine constructing an entirely new ecosystem just to allow people to pretend modules are something they're not. This is not just overkill; it's a preposterous reinvention of a wheel that insists on being a pentagon!

Sure, if devs want to globally expose their variables, entwining everything in a static context, so be it. BUT, why should we, who walk the righteous path of dependency injection, be subjugated to this configured chaos?!

My blood boils as the jestering Jest thrusts upon me a fragile, perpetually breaking system, punishing ME for its determination to support whole module mocking! A technique, mind you, that I wouldn’t touch with a ten-foot pole, because, you know, DEPENDENCY INJECTION!

Where are the alternatives, you ask? Drowned in the abyss, it seems! Why can’t we embrace snapshots and all the delectable integrations WITHOUT being dragged through this module-mocking mire? Can’t module mocking just be a friendly sidekick, an OPTIONAL add-on, rather than the cruel dictator forcing its agenda upon our code?

Punish those clinging to their static contexts, their global variables – NOT those of us advocating for cleaner, more stable practices!

It’s high time we decouple the goodness of Jest from its built-in bad practices. Must we continue to dance with the devil to delight in the depth of Jest’s capabilities?

WHY, Jest, WHY?! 😭

I think another intriguing job asides programming is engineering (*for some*). A week has past and I've been on the hike assisting my beloved brother on his contracted engineering job while I am less occupied. The job is based on 🗼Tower analysis and It's quite risky as you'd have to climb up to 56 meters high just to take readings of antennas, and fix some other stuffs. The only thing I find intriguing about this job is his love for it, funny enough he also thinks I love the job too and I guess I'm guilty for his thoughts (*Sorry bro, I love the job for you not me*).
With my little experience so far on my *new brotherly job* I noticed the most hectic task isn't going up and down the tower taking readings but at the end of all operations, he'll have to gather the values and snapshots he took while on the tower to prepare reports on msword & excel for the other buttwags at the office (or home I guess)
then archive and sends via mail. Seeing this lengthy process I was forced to ask why he wasn't using any reporting tool like Jotforms or any other equivalent and I was willing to look up some recommendations for him, his reply was: "I'm already used to this form of reporting, its what I was trained with and what the company provided, nevertheless a friend of mine suggested something of such weeks back but I would have to pay monthly fee for its usage which is quite on the high side and I don't think I'd prefer that."
Sounds convincing but not enough, okay here is another deal: You use an android phone right? and at my office we work on system automation (*basically does not know what I do for a living probably thinks I'm a hacker the illegal one*), how about i design you an android app for you to capture the tower data and a PC software for you to auto generate the msword & excel reports, I can get this ready for you in less than 5 nights (*I've got less task on my desk, and was willing to take the timeout to prepare the solution that he needed, all I needed to hear for a kick start was an "Okay" just to be sure he wants it*) I suggested and re-assured but up to this point he still declined my offer and is willing to stick with his current reporting pattern (*Me died*).

A peace of work to be done within 2 weeks for 3 devs
1 dev gets pulled off the team, the other decides now is a good time to take a 2 week holiday
So am the only one left and because the designs weren't even ready for the first week i only had a week to do it, i managed to finish it with some defect obviously, and testing oh yes the testing non, non whatsoever. Only test i had were snapshots. Other than that, nothing. So the demo seemed to please everyone and the whole team got praised for some gr8 work, work that was estimated for 3 but done by one and hah no testing...yet

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

Ever given access to production data to correct a data entry that doesn't have a PATCH api ready?

Ever ran UPDATE table set COLUMN="value"; without WHERE condition?

Ever done both of these sequentially?

Yeah. DB snapshots cost a bomb to restore :(

Being able to summon CTRL+Z's and/or save timestamped Universe state snapshots.

I'm easily a Digital Oceans fans, though I have heard horror stories, so I might set up a system to do regular backups.

I'm considering migrating my current server to something FreeBSD based, so I can easily do ZFS snapshots, and even code on my machine at home and just send the jail as a snapshot. Like docker, but different.

I wonder one day whether we will br able to take snapshots of the internet given the amount of bandwith vs the storage at out disposal... reason from my side will be to see the difference correlated over time especially with regards to the "legal crap". Also having the ability to see other people's "copy" overlayed with yours to see the actual organic nature of information.

I just finished replacing perl's builtin Storable with my own deranged take on it because I wanted to be more efficient about saving and loading snapshots of a VM to disk.

It was a resounding success, of course. But what am I doing with my life?

Does anybody how to export Virtualbox Linked Clones and import them on a different machine withlut error?

With brute copy & paste, the linked clones can not be started on the ither machine.

I know that I can export the Appliance as an .ovf file, but this does not save snapshots of the machines.

tl;dr: azure support are utter bollocks

so about late june-ish, my azure student subscription expired, which i wasn't notified about. but that's fine, surely once it's expired i can get my data back, right?

...right?

i try to download the .vhd file with my nodejs project on, and then contact their support after failing to mount the vhd. i asked them whether they could get my data for me (or at least provide some clear instructions, in case i mounted the vhd incorrectly). instead i was told to do loads of things, creating blobs, making snapshots, etc... all of which did absolutely nothing.

mid-august, i'm still trying to get my data back, when i get a call from, you guessed it, microsoft azure. a manager had told me that all my data had been lost, and that i was eligible for $500 in credit in compensation. i was angry (and rightly so), and refused their offer. i emailed azure support again expressing my anger, for them to tell me that my data wasn't lost...?

come to mid-september, and and i was fed up of waiting for my project. i wanted to finalise the fucker and launch the website, but azure had stalled me for well over two months. i had to put some money towards azure just to start up the vps, zip up the project, transfer it to another vps, and shut it back down.

and that kids, is why i wouldn't ever recommend azure.

ps: yes, i'm backing up files daily from now on

So... I've been thinking, I tend to default to LVM when trying to create easy-to-manage disk partitions, or when I want to backup a database without long lockings during a dump... Though, now... I got thinking.

What do you guys think, which is better in terms of functionality: BtrFS or LVM?

I know BtrFS offers such thing like full snapshots that allow to easily transfer just the increment over the snapshot origin off to a remote server for archival, but I never fully grew to trust btrfs as a server filesystem... Its...

Younger, and not as widespread, not to mention I don't know any performance statistics to recommend its use for this or that case (Like... Would a high-load database engine stutter flushing all those changes on disk while reading / writing temp tables and such)

I'm trying out some stuff I read. I have rich domain models with private fields. In order to create EF core entities my domain models create snapshots with public properties.

So 3 models - domain, snapshot, entity.

Now I am thinking about introducing a fourth for the API.

This seems mental and that I have misunderstood something. Automapper might help clean this up.