Saying Thank You

I chose to share knowledge with you for free, without ads and without a pay-wall. I'm grateful to the few people with hearts of gold, who at least stop by to say thanks for the amount of effort I put in to help save your time and effort.

• You could convey your thanks here: https://forms.gle/aw3EzLVnLVSjSm2B8

and/or

• Send a gift from my wishlist via your Amazon account: https://www.amazon.in/hz/wishlist/ls/2V4F7N8K7CYX5?ref_=wl_share

 

When using the Hemmingway Editor, take the grading with a pinch of salt

For writing assignments, many people recommend the free Hemmingway editor, which help remove unhelpful adverbs, passive voice and in general, helps improve your sentences by making them simpler and easier to read. These features are really useful.

However, recently when I wrote some simple sentences, the editor said they were at Grade 11, when these were simple sentences that even a 7th or 8th grader could understand. So I did a little experiment. I chose sentences from two children's books and checked to see what Hemmingway editor would report.

First book: The call of the wild.

This is a version simplified for children, meant for 3rd, 4th or 5th graders. While most paragraphs showed a “Grade 5” or “Grade 6”, a sentence whose only fault was that it was too long, showed a “Grade 7” difficulty.

Moreover, copying and pasting the same sentence repeatedly, brought the score up to Grade 11. From there, it was not difficult taking it up to Grade 13. This is a serious flaw in the algorithm.

Second book: Chips (story of a rabbit).

This book has very simple sentences meant for children in the 2nd or 3rd grade. While most paragraphs showed grades ranging from “Grade 5” to “Grade 9”, some sentences gave these results:

Oddly, removing a few sentences brought it to Grade 10.

Removing more sentences pushed it to Grade 11.

By repeating the same sentence, I could push it to Grade 14.

I've written to the creators of the editor, showing them these examples. Hope they'd correct it soon. Meanwhile, writers should take the gradation score with a pinch of salt. The editor is still a good method of evaluating your text for the other features it provides.

Repairing grub in a dual boot system without using Boot Repair

After installing Windows, installing Ubuntu detects Windows and installs fine, giving you a dual boot system. But if you re-install Windows or install another OS on another partition, chances are you'll get a MokManager error that looks like this:

Before attempting to repair Grub2, go to your BIOS (as shown in this page) and select the grubx64.efi file or whichever efi file is relevant to your installation, and reboot. Chances are that the non-booting is just being caused because the computer does not know which efi file is safe. I think this happens due to secure boot being enabled, but please don't disable secure boot. Just specifying the correct efi file may help.

If it still isn't booting and you want to repair Grub2, follow the steps in this or this website. I had tried using Boot Repair, which worked nicely, but only once. So I now prefer doing it manually.

Steps:
1. Boot from a live CD or live USB which has the same Ubuntu version as your latest installed Ubuntu.
2. Use sudo fdisk -l and sudo blkid or GParted to determine which sda Ubuntu is installed on. Let's assume it's on sda4.
3. sudo mount /dev/sda4 /mnt
4. for i in /sys /proc /run /dev; do sudo mount --bind "$i" "/mnt$i"; done
5. If Ubuntu is on an efi partition (check using sudo fdisk -l | grep -i efi), mount it with sudo mount /dev/sda4 /mnt/boot/efi.
6. This command won't work if you are using a 32 bit live boot disk on a 64 bit computer, so make sure you use a 64 bit live boot disk: sudo chroot /mnt
7. update-grub (if this doesn't work, goto step 8. Else, step 10).
8. grub-install /dev/sda
9. update-grub. To locate and add Windows to grub menu.
10. If Ubuntu was installed in efi mode, run this: blkid | grep -i efi; grep -i efi /etc/fstab
11. Reboot the computer.

Even now, the "Failed to start MokManager" error may show up. Just go to BIOS and select the right efi file and reboot. Everything should work fine now.

 

Update: If the bootloaders are on separate disks, I've mentioned how to fix it here.

Removing malware from PDF's

Sometimes PDF files can contain malware. Although the techniques listed below are not a fool-proof guarantee of getting rid of malware, it can get rid of malware that's embedded in the metadata of PDF files.

First, confirm that the PDF has malware.

Now create the following bash script in a file named malwareRemover.sh:
(make sure you use a tab instead of spaces for the lines within the for loop).

#!/bin/bash
clear
for f in *.pdf; do
    echo "Removing malware from $f:"
    echo "-------------------------------"
    echo ""
    #---convert
    tempFilename="temp.ps"
    echo "Converting $f to $tempFilename"
    pdf2ps $f $tempFilename
    echo "Deleting $f"
    rm $f
    echo "Converting $tempFilename back to $f"
    ps2pdf $tempFilename $f
    echo "Deleting the ps file"
    rm $tempFilename
done

Change the permissions of the file and install ghostscript:
chmod +x malwareRemover.sh
sudo apt-get install ghostscript

Run it with all the PDF files in the same folder as malwareRemover.sh.
./malwareRemover.sh


If you are using Didier Stevens' tools, this script will help:

#!/bin/bash
clear
for f in *.pdf; do  
    echo "Scanning $f for malicious data:"
    echo "-------------------------------"
    echo ""
    #---scan
    python3 pdfid.py $f
done

echo ""
echo "How to interpret the info:"
echo "------------------------------"
echo "* Almost every PDF documents will contain the first 7 words (obj through startxref), and to a lesser extent stream and endstream. I’ve found a couple of PDF documents without xref or trailer, but these are rare (BTW, this is not an indication of a malicious PDF document)."
echo "* /Page gives an indication of the number of pages in the PDF document. Most malicious PDF document have only one page."
echo "* /Encrypt indicates that the PDF document has DRM or needs a password to be read."
echo "* /ObjStm counts the number of object streams. An object stream is a stream object that can contain other objects, and can therefor be used to obfuscate objects (by using different filters)."
echo "* /JS and /JavaScript indicate that the PDF document contains JavaScript. Almost all malicious PDF documents that I’ve found in the wild contain JavaScript (to exploit a JavaScript vulnerability and/or to execute a heap spray). Of course, you can also find JavaScript in PDF documents without malicious intend."
echo "* /AA and /OpenAction indicate an automatic action to be performed when the page/document is viewed. All malicious PDF documents with JavaScript I’ve seen in the wild had an automatic action to launch the JavaScript without user interaction."
echo "* The combination of automatic action  and JavaScript makes a PDF document very suspicious."
echo "* /JBIG2Decode indicates if the PDF document uses JBIG2 compression. This is not necessarily and indication of a malicious PDF document, but requires further investigation."
echo "* /RichMedia is for embedded Flash."
echo "* /Launch counts launch actions."
echo "* /XFA is for XML Forms Architecture."
echo "* A number that appears between parentheses after the counter represents the number of obfuscated occurrences. For example, /JBIG2Decode 1(1) tells you that the PDF document contains the name /JBIG2Decode and that it was obfuscated (using hexcodes, e.g. /JBIG#32Decode)."
echo "* BTW, all the counters can be skewed if the PDF document is saved with incremental updates."
echo ""
 

Which conferences and journals allow pre prints on ArXiv and which don't?

ArXiv, TechRxiv (owned by IEEE) and HAL are pre-print repositories where scientific papers can be submitted as proof of precedence, so that the ideas do not get scooped. Especially when you are submitting a paper to a journal, it's better to put it on a pre-print repository, because the journal's review process can take very long and the process of making corrections and resubmitting can take many years until the paper finally gets published.
But there may be conferences or journals which may not allow authors to publish their work on pre-print repositories, because it would interfere with the double-blind review process.


These are conferences/journals which allow authors to publish on ArXiv:

• IEEE allows it. They have some formal policies about it.
• Elsevier allows it. 
• Springer allows it.
  

These are conferences/journals that don't allow it:

• KDD. Prior to 2019, KDD used to allow pre-prints on ArXiv.

I hope to update this page with more names of conferences and journals. If you know of any, please mention it in the comments.

GMail's autocorrect messing up your email?

We don't allow children to drive cars, but somehow people think it's ok to allow a self-driving car on the road...an algorithm that has no clue about what the world is and no clue about what driving is. AI has a long way to go before becoming intelligent.

Similarly, GMail tended to autocorrect our words for us, without even highlighting the words it autocorrected. The result? I typed "presenter" in a sentence and in the next sentence when I typed "presenting", GMail discreetly autocorrected it to "presenter", giving the sentence a grammatical error which gave a bad impression to the reader. I noticed the error only after sending the email. The minimum GMail could have done was to highlight the word that had been autocorrected or to ask for my consent before introducing the feature. Google seems to need better engineers.

When I first heard that a person lost his job because of autocorrect correcting a word to "hash brown" when he emailed a colleague, I didn't actually believe it happened unintentionally. But after seeing how autocorrect messed up my email, that man has all my sympathies.

There are many such examples. Some of these may be fictional, but they do drive home the point.

Autocorrect can be switched off in GMail in the General Settings. I recommend switching off all of the AI features that aren't reliable.

IISc Open Day 29th Feb 2020

The Indian Institute of Science's Open Day is a yearly event during which the general public can visit the campus free of cost and every department exhibits some fascinating experiments or presents the research that students are doing there. Most of these are at the cutting edge of technology, so it's an excellent learning experience for everyone. The campus is so vast that even for someone arriving at 9am, it's hard to visit all the stalls and view all demo's and experiments. Of the many demo's, here are some:

Also known as Magic Mud or Oobleck, the video I took, shows how a person who inserts their hand slowly into the liquid is able to, but the person who jumps on it does not sink. The liquid is just a 1:1.25 ratio mixture of water and corn-starch. The corn-starch doesn't dissolve in the water, and stays suspended as a non-Newtonian fluid. If you are gentle, the liquid parts and allows solids to enter. If you punch it or run over it, if feels like a solid object. I took a thin, sturdy leaf stalk and thrust it into the liquid, but it wasn't able to penetrate. So surface area didn't matter. I'd like to have seen if a knife edge's surface area would be able to penetrate. When I suggested it could be used for bullet-proofing, the organisers said that's one of the applications considered for this fluid. I jumped on it, ran on it and even punched it. Solid as a rock! Be gentle, and it's cooperative. Made me realise that many people behave like non-Newtonian fluids too. :-)

Studies are being conducted on the structure and role of certain proteins which play a role in how ambient light affects the body's sleep cycle.

This was an actual brain-machine interface demo where the person wearing a skull-cap that noted his brain's gamma waves, was able to control a circle on the screen and make it move left or right just by using his thoughts.

Hardware relics from the past.

The presence of such ponds and greenery in an institute campus speaks a lot about the quality and care that goes into maintaining it.

This is a pacman game where they train a deep neural network using the visitor's hand gestures that indicate up, down, left or right motions. The visitor then uses these gestures to move pacman on the screen and play the game of escaping the ghosts. Cumbersome, but interesting.

A functional hydroponics setup that's 90% more water-efficient than traditional farming. The nutrient-enhanced water mist gives nutrition to the roots.

Just a car that follows a trail using infra-red light.

Research being done on mice to understand how the hippocampus stores information on spatial recognition. The brain has grid cells, head direction cells, boundary detection cells etc.

A working demo of the Tesla tower. The CFL bulb glows due to wireless electricity (ionization).

A demonstration of how Tsunami waves are formed, how the wave appears to recede just before the main wave strikes and how barriers may help push the wave backward to cancel out incoming waves; demonstrated by a guy who appears to be infected with the Corona virus :-).

One of the e-rickshaws at IISc that the public could use for free. They are labelled "Transvahan", which initially gets interpreted as "Transformer vahan", but don't worry; it's not going to turn into Optimus Prime. It's actually "Transport vahan". There's very few of these vehicles, and most of the time I saw them transporting water and food. It would've been nice if there would be plenty of bicycles that'd help traverse the vast expanse of the campus during such events. They did a really good job of arranging for plenty of good quality drinking water sources all over the campus.

There were so many many more stalls and demo's that I didn't mention just for the sake of brevity. Overall, a really good experience. Open Day is a must-visit for science-lovers.

Configuring LiClipse to run Anaconda or Miniconda

I'll have to admit, it is indeed better to use a virtual environment when dealing with Python. If you mess up your native Python, the system won't work properly. But the problem with using Python this way, is that there may be packages that aren't available for install via conda for particular versions of Python. That gets very annoying.
This article explains how to setup LiClipse to work with Anaconda or Miniconda. Specifically, Miniconda.

Open LiClipse. In the main menu, goto Window > Preferences.
Type "interpreters" in the search box.
Click "Python interpreter".
Click the "Browse for Python/pypy exe" button.
Close the first window that opens and you'll see this:

Browse to your Miniconda folder and select the python3 or python file in ~/miniconda3/bin.

In Interpreter Name, enter a name of your choice: MinicondaPython3.7, for example.

One more window opens up. Click Ok.

Now (assuming your Python project is open), in the main menu of LiClipse, select Run > Run Configurations.
In the left column, right click on Python Run. Select "New Configuration".

Under the "main" tab, choose your project name and specify the main file.
In the "interpreter" tab, select MinicondaPython3.7.

That's it. Install any necessary Python packages using the conda command, and you are ready to run your project!

Clearing swap space in Ubuntu

When having two large applications open in Ubuntu can cause slow performance because of swapping memory from hard disk to RAM, it leaves one perplexed to see swap space still being used after all applications are closed.

I investigated this a bit more to find out that it's better not to mess around with the kernel's swappiness parameter, and that using the "top" program or other scripts is not really necessary. You don't even need to use vmstat to check the si and so columns to see if swaps-in and swaps-out are happening.

There's a far simpler option that promptly copies all swap memory back to RAM.

sudo swapoff -a; sudo swapon -a

I opened a large application by the time I took the screenshot, hence the greater usage

Enjoy!

Converting webp files to more familiar formats in Ubuntu

With Google's efforts at better image compression, came an acquisition that gave rise to the webp image format. Although it was announced in 2010, I was surprised that Ubuntu 16.04 didn't support it natively.
To use it with Gimp, you need one of the latest versions. Good news is, ImageMagik supports it if you install webp. Once installed, you can batch convert your webp files to png too. Two simple steps.

To install:
sudo apt-get install webp

To convert all webp files in a directory to png:
find ./ -name "*.webp" -exec dwebp {} -o {}.png \;

To delete the original webp files after converting to png (though I'm sure there has to be a better way):
find ./ -name "*.webp" -exec dwebp {} -o {}.png \;; find . -type f -name '*.webp' -delete

The precious thing frequently stolen from you: Sleep

More than seven years of noting observations and trying to correlate the causes of severe eye strain and poor vision, led me to the root cause: The lack of uninterrupted deep sleep for seven to eight hours every night. The funny part is, that if you ask anyone if they are getting enough sleep, they'll immediately say "yes", even though they are actually getting only around four or six hours of bad sleep. Even I said "Yes" to the first doctor who asked if I was getting sufficient sleep (when my eye strain was at its severest and I was getting only 4 hours sleep).

There are bodily repair processes that do not get completed if deep sleep is interrupted before 8 hours, and the debt accumulates over the years. I've confirmed this multiple times. There's a huge difference between "resting" and "8 hours uninterrupted sleep". I have good reason to say it's the lack of sleep and lack of proper nutrition that end up giving people weak vision and a lot of other health issues. Six hours or four hours sleep is not enough. The people who have been insisting for all these decades about seven to eight hours, were always correct.

Did these affect you?

• Children being woken up early for school.
• Children being told to stay awake late or wake up early to study during exams.
• Children staying up late to watch TV or play video games etc.
• Adults being forced to work late due to deadlines.
• Having to use an alarm clock to wake and not be late for work or to cook breakfast.
• Doctors being woken up for emergency cases.
• Family members of patients being woken up during a hospital stay.
• Taxi drivers and auto-rickshaw drivers losing sleep to make ends meet.
• Phone calls and extraneous noises when sleeping.
• Health issues that cause people to wake up once or more at night.
• Improperly cooked food or adulterants or burnt particles in food affecting the digestive system and causing sleep loss.

Sleep Requirements researched by the National Sleep Foundation.

Is there anything we can do to change the culture of sleep loss?

• Insist on properly cooked, unadulterated, un-burnt, healthy food and water at home, canteens and restaurants.
• Allow for flexible work cultures and work-from-home options.
• Allow schools, colleges and offices to begin at 11am for some people (night-birds) and at 9am for other people (larks). 
• Spread awareness of this at school level and get a buy-in from parents and employers.
• Arrange for better work shifts and recruit more people to handle workload.

We need to create ways to make this happen in a cut-throat rat-race. We've been through centuries of brainwashing about the heroics of equating hard-work with not sleeping. "How many people's health are you permanently willing to ruin in order to make a living?". Losing sleep even occasionally is not good. Don't allow people to nudge you into it. Don't nudge children into it.

There's no heroism in losing sleep.

Healing eye strain: https://nrecursions.blogspot.com/2020/11/the-real-cure-for-eye-strain.html

Copy pasting and clearing screen in the Ubuntu terminal

Copy pasting with Ctrl+c and Ctrl+v works fine on the MacOS terminal, but doesn't work in Ubuntu. I assumed this may be some security feature or because Ctrl+c is the age-old command for an interrupt, so I didn't give it much thought and just used the mouse to copy paste.

But today it just struck me that I should find out the real reason, and viola! It's just a matter of pressing Shift.

To copy:
Shift+Ctrl+c

To paste:
Shift+Ctrl+v

and as a bonus:
To clear screen on the terminal:
Ctrl+L

This clear screen also supposedly works in Matlab, MySQL etc., though I can confirm it works on the Python terminal.

Aquaplaning, and how to avoid it

Long back, I was curious about why Formula One racing cars had such "smooth" tyres. The racing slicks. Turns out, when there's no water on the road, it's actually more beneficial for more of the tyre to make contact with the road and provide more grip. Having grooves would lessen the grip. When the road is wet however, the grooves/treads on the tyres are absolutely necessary to prevent aquaplaning. The rain tyres allow water on the road to enter the grooves, thus allowing the remaining part of the tyre to make contact with the road and give better grip. Without this, the car would go completely out of control.

Exactly the same principle is used when designing footwear. The process of cutting such slits is called Siping. Recently, I purchased a new pair of rubber slippers and noticed that it didn't give any grip on wet surfaces. I even slipped and fell down once.

On examining the sole, I realized why.

No grooves.

So I took a knife and cut out some crude grooves all along the front, mid and back portions of the slippers. Just these crude grooves were enough to give a good grip on wet surfaces. I also wrote to the company, chiding them for not considering this basic design element. They wrote back thanking me for the valuable feedback and stating that they were the number one footwear brand in the country. The number one brand? With a safety aspect like this disregarded, I definitely don't consider them number one, but I'd be willing to consider it if they make the necessary design changes.

Update: The horizontal groove I made was a mistake. When the slippers flex when walking, the groove deepens and causes the rubber to break at that point. So the better option is to either cut grooves vertically or just buy new slippers with better soles.

Realtime plotting in Python

If you intend to create plots/graphs with Python, your first choice may be Matplotlib. Don't use it unless your plots are extremely simple and isn't expected to grow more demanding with time. I used Matplotlib for realtime graph plotting for a really tiny graph and it was disappointingly slow. Moreover, it required installation of Python 3.6, which wasn't available readily for Ubuntu 16, and caused even more problems.

Then I came across PyQtGraph, and was impressed by the super-fast plotting speed and the mind-blowingly impressive variety of graphing functionalities. Really. You gotta see this.

Install it using: pip install pyqtgraph
Run the examples with: python3 -m pyqtgraph.examples
And prepare to be impressed!!!

I've come to understand that the rendering is quick because the underlying code is in C/C++ and the Python bindings are merely meant for invoking that code.


Still, even though the graphs are very impressive, the non-linearity of the code examples and the lack of code comments make it really hard for a newbie to understand how to get a basic graph running.

Here's where I help. After spending an extraordinarily long time figuring it out on my own, I've created a simple, basic example with which you can create a realtime graph. Hope it helps...


import time
import random
import pyqtgraph as pg
from collections import deque
from pyqtgraph.Qt import QtGui, QtCore

class Graph:
    def __init__(self, ):
        self.dat = deque()
        self.maxLen = 50#max number of data points to show on graph
        self.app = QtGui.QApplication([])
        self.win = pg.GraphicsWindow()
       
        self.p1 = self.win.addPlot(colspan=2)
        self.win.nextRow()
        self.p2 = self.win.addPlot(colspan=2)
        self.win.nextRow()
        self.p3 = self.win.addPlot(colspan=2)
       
        self.curve1 = self.p1.plot()
        self.curve2 = self.p2.plot()
        self.curve3 = self.p3.plot()
       
        graphUpdateSpeedMs = 50
        timer = QtCore.QTimer()#to create a thread that calls a function at intervals
        timer.timeout.connect(self.update)#the update function keeps getting called at intervals
        timer.start(graphUpdateSpeedMs)   
        QtGui.QApplication.instance().exec_()
       
    def update(self):
        if len(self.dat) > self.maxLen:
            self.dat.popleft() #remove oldest
        self.dat.append(random.randint(0,100)); 

        self.curve1.setData(self.dat)
        self.curve2.setData(self.dat)
        self.curve3.setData(self.dat)
        self.app.processEvents()  
       

if __name__ == '__main__':
    g = Graph()
   
    

 

________________________________

The fact that you are here means that you use the computer a lot. So please also take some time to understand the real cure for eye strain. Yes, it matters.

Unable to signout / logout of GMail?

The fact that GMail used via the mobile web browser has such a silly bug that didn't get fixed even after I reported it thrice, is a sad state of affairs.

Anyway, if the signout menu isn't appearing, simply type "google.com" in the address bar and you'll see a signout button at the top right corner.

That's it!