Changing the colors in Pandas boxplot

 Normally, it's just one Google search that gets you to a good example for plotting any graph using Pandas or Matplotlib. Not so for boxplots. Every example I saw was ordinary and didn't show the advanced options for colouring the plot. So here's a quick demo:

import math
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
#Named colors: https://matplotlib.org/3.1.0/gallery/color/named_colors.html

data = {"A": [1,2,3,4,10,56,32], "B": [2,4,6,3,2.4,0,0], "C": [1,4,0,10,20,5,2], "D": [2,3,6,2,3,40,-1]}
for k in data.keys():#convert the zeroes to NaN, so that it does not mess up the boxplot
    data[k] = [x if x != 0 else math.nan for x in data[k]]
figWidth = 4; figHeight = 4.1;
boxproperties = dict(linestyle='-', linewidth=1.5)
medianproperties = dict(linestyle='-', linewidth=1.5)
flierproperties = dict(markerfacecolor = 'lightgrey', markeredgecolor = 'lightgrey')
colors = dict(boxes='dimgrey', whiskers='darkgray', medians='yellowgreen', caps='dimgrey')
df = pd.DataFrame(data, columns=list(sorted(data.keys())))
df.plot.box(grid=False, patch_artist=True, showmeans = False,
            sym='.', color = colors, boxprops=boxproperties,
            medianprops=medianproperties, flierprops = flierproperties,
            figsize=(figWidth, figHeight))
plt.title("values")
plt.ylabel("y label")
plt.xlabel("x label")               
plt.savefig('myPlot.eps', format='eps', dpi=1200)
plt.show()   

There are of course more styles that could be presented. Do let me know in the comments if you'd like me to add them or if you have any improvements to suggest.

What technical pre print repositories accept unpublished research?

In academia, it is very important to be ethical and to ensure your ideas are not copied or scooped before you can prove that you came up with the idea first. To enable this, pre-print repositories like ArXiv were setup to give research work a timestamp when submitted. This could help any researcher prove that they came up with an idea first. Any work found on a pre-print repository is not a credible source of information, since it has not yet been peer-reviewed. However, since ArXiv has a very strict moderation policy that results even in experienced scholars having their papers rejected, ArXiv does not fulfill the purpose of being a pre-print archive. The delay in moderation, the need for endorsement from somebody else and the rejection can just end up allowing time delays that can result in work getting scooped. Many other pre-print repositories have come up, and it's considered ok to upload your research paper to multiple repositories.

Some of these free repositories are:

• ViXra: This pre-print repository is far more accepting than ArXiv, and was created as a means of allowing genuine research work to get a timestamp. I found their FAQ to be far more insightful and enlightening than any other pre-print's FAQ. I encourage reading their FAQ. It mentions a lot of useful information about what you should and should not do with your pre-prints. I feel ViXra is one of the only pre-print repositories that actually honors the purpose of being a pre-print repository. Hence, this is the only repository I would trust, because I know that when I submit a genuine paper to them, they will give me a timestamp for my work instead of delaying it or rejecting it.
• TechRxiv: A pre-print repository setup by IEEE. I've found them to be good, but there are chances of a reviewer not accepting a paper and having to appeal to the editor until they allow a re-upload. The whole process can take a few weeks...enough time for getting scooped in some fields.
  
• IndiaRxiv: I was pleasantly surprised to find one for India. It's part of OSF, which accepts papers from multiple disciplines.
• HAL: This is a well known repository with moderation and it is said that they also allow you to transfer your paper from HAL to ArXiv.
• EngrXiv: This is also part of OSF, and has been gaining traction with researchers.

Some precautions before submitting to a pre-print
 
Remember that there are some journals and conferences that do a double-blind review of your paper, and they will not accept a paper that is uploaded to a pre-print repository. so if you plan to submit to a certain journal or conference, first check their website or write to them asking if they allow papers that are on a pre-print repository.
A quick way to search is by using Google's tools:

site:journalWebsiteName.com arxiv

Careful of predatory journals or conferences

Some journals will offer to get your paper published for free without any peer review process. Avoid such journals. It can lower your academic standing. I was contacted by one such journal which quoted an ISSN number on their website. When I searched, I realized that such an ISSN number didn't even exist!!! Always make sure your work is properly peer reviewed by reputable journals. In the tech world, journals with a Scopus index are safe.


In conclusion
 
Writing research papers is a time-consuming and cumbersome process. All that hard-work should not go for a waste. Pre-print repositories are a very useful means of proving precedence of your idea. They are also meant for inviting comments from colleagues, before you decide to submit it to a conference or journal. Websites like ResearchGate will automatically detect your work on pre-print repositories and link it to your ResearchGate account, making it easy to share your work and to invite comments on ResearchGate, because it automatically notifies the authors you have cited in your paper.
To simplify people's research process and to make available information on research and publications, I've curated a small collection of helpful documents from multiple people.


ps: On a more personal note, I feel that researchers who advance the knowledge of mankind and who are the backbone of the technological capability of society, need more respect and recognition in society. The level of detail they go into, the expertise they acquire in each subject and the rigor of research makes them an elite force. It's unfair that companies and publishers make profits form their ideas but the researchers don't profit from it. How did researchers even agree to such a raw deal? Publish or perish? I dearly wish that the world figures out a way to ensure that researchers are able to monetize their contributions much better than they currently are. If not that, then at least provide them privileges and conveniences that make life easier for them. All the benefits of technology we enjoy are because of them. We must never forget the shoulders we stand on.

The better way to calculate permutations programmatically

When wanting to calculate factorials and permutations, I was surprised that C++ didn't have a built-in function to do it. Sure, it's a simple function, but cmath could have had one instead of expecting users to write their own function. Anyway, I created a class for it and soon realized that even unsigned long long was unable to hold the large numbers that I was calculating permutations for.

This led me to the online calculators, which did ok for conditions like 81 numbers taken 81 at a time (which gives a huge result: 5797126020747367985879734231578109105412357244731625958745865049716390179693892056256184534249745940480000000000000000000).

However, calculations like 129600 taken 2 at a time, were not supported, so when I searched for which programming language could help, I found that Python was just perfect for this. Turns out, numbers in Python are only limited by the amount of RAM you have. Intelligent people, these people who maintain Python. 

Python does have a limit on how deep your recursion can go, so rather than use recursion to calculate factorials, it's better to use the humble for loop. Here's how it's done:

def factorial(n):
    fac = 1
    if n == 0:
        fac = 1
    else:
        for i in range(n,0,-1):
            fac = fac * i
    return fac

print("To calculate permutations:")
print("n!/(n-r)!")
n = int(input("Enter number of possibilities (n): "))
r = int(input("Enter how many at a time (r): "))
print("Permutations: ", factorial(n)/factorial(n-r))

The Real Cure for Eye Strain

When you wear a pair of shoes that’s too tight for you, and each time you walk, it keeps hurting you, would you remove it or would you pour medicines and apply band-aid on your feet and continue walking while the shoe bites into your skin and flesh? The shoes are an analogy to the way we use our eyes without taking rest. The band-aid and medicines are an analogy to how we use spectacles, eye drops, eye exercises, blue light filters etc., hoping that it would cure us (Blue light blocking lenses have shown no evidence of preventing digital eye strain. Ref1, Ref2.).

Sure, some remedies do appear to give a little relief, but they are only treating the symptoms. We should not pretend that it is working. We need to treat the root cause and cure it. I realised after a decade of eye strain, that there's an obvious, common-sense first-step to the healing process: Removing those "shoes".

Discovering the truth

Doctors tend to focus on intraocular muscles. A less considered fact is how extraocular muscles (shown in red) are affected. I’ve felt excruciating pain in those areas for many years.

In my case, it began with a burning feeling at the front of my eyeballs. Then came the occasional sharp pains at the sides of the eyeballs, which became sharp stabbing pains all around and behind my eyeballs. I ignored that too and continued working. Then came severe pain all over the eyes, eye muscles twitching, tightness at the cheek muscles that even spread to the forehead muscles. During the next few years, I consulted plenty of doctors (even at highly recommended eye speciality hospitals). They examined my eyes and found no issue other than short-sightedness. They tried their genuine best to help, and I could see they felt bad, because they wanted to help me recover (bless them), but they didn’t know how to cure me. Medical science doesn’t seem to have given enough attention to the root causes of fatigue-related eye strain. Practitioners of ancient traditional medicine couldn’t help either. The persistent pain finally drove me to note down daily experiences. Over a period of few years, multiple experiments, correlation and keen observation, I finally found the cure and started recovering.

The Cure

It’s necessary to follow these three very strictly. Depending on the level of strain, it can take anything from a few weeks to many months to get cured, but this is the only way to cure fatigue-related eye strain. Nothing else works.

1. Eight hours of uninterrupted sleep every night: This is the foundation. Without this, nothing else will cure the strain. We assume we are getting enough sleep because we feel refreshed when we wake up…and then don’t notice that we still feel sleepy during the day. Before my eye strain got bad, I hadn’t noticed I was getting only 4 hours of sleep for many months. During my experiments, I noted that it was only after 8 hours of uninterrupted sleep, that my eyes felt fine. Interruptions in sleep or short duration of sleep do not seem to allow the muscles and tissues to heal. Strain appears to accumulate everyday due to lack of proper sleep. You’ll also notice that you are able to see much clearer than usual, after you get 8 hours of deep sleep. Children need more than 8 hours of sleep, and noise etc. disrupts sleep.
2. Closing the eyes after 20 minutes of strenuous work: Twenty minutes of work followed by closing the eyes, relaxing the face muscles and neck muscles and waiting for a while until the strain subsides (the way we maintain tension in our muscles is a small part of the problem). Although I don’t recommend longer periods, some people may need 45 minutes to 1 hour of continuous work (provided they get sufficient protein in their diet), but more than an hour is harmful. Even if you don’t feel strained, stop after an hour and close your eyes. It’s very necessary. I’ve tried eye drops, an eye gel, anti-reflective coatings, blue-light filters, eye exercises, washing the eyes, the 20–20–20 rule, expensive eye glasses and all the other advice that doctors and common folk think is a cure for eye strain. Nothing worked. At my level of severity, even minor differences were noticeable. The ONLY way to rest the eyes and get relief from fatigue-related strain, is to keep the eyes closed until the strain subsides.
3. A well balanced diet of properly cooked food: It’s extremely important to realise that when food is not properly cooked or even slightly burnt, it causes discomfort in the digestive system (most of the time you won’t even notice this discomfort) that leads to sleep loss (I’ve mentioned more about this below). Also, ensure you search the internet and understand what a “well balanced diet” means, and why it’s important. Instead of searching only for foods that can help the eyes, make sure your entire body gets the nutrition it needs. Particularly, when muscles go through strenuous activity, they need a proportional amount of protein (there’s a wide variety of proteins, by the way). The body also needs natural foods that help with long duration sleep and deep sleep.

Eye strain and myopia can be caused by many other factors too. Those are best diagnosed and treated by a qualified ophthalmologist. Everything I’ve mentioned here, is only for eye strain that’s caused by excess fatigue to the eyes and lack of sleep.

 

Precautions and Observations

• Eye exercises: Tired muscles need rest. Not exercise. When your eyes are strained, do not do eye exercises. Just close the eyes to rest them.
• Full body exercises: It was of no use in curing or reducing eye strain, and it resulted in better sleep only during the first week. However, full body exercises are very necessary for general health.
• The “SCREAMING”: The pain/burning/discomfort is basically your eyes screaming at you, begging for rest. Never ignore or delay it.
• Digital screen settings: It helped a little bit when I reduced my computer and smartphone brightness to the lowest level I was comfortable with, and used the monitor’s preset “warm” setting. Alternatively, the free night-filter apps are also good. What also helped, was positioning the screens in such a way that reflections on the screen did not cause glare. Anti-glare coatings did not reduce my strain even a little bit because digital screens or blue light was not the reason for strain. Strain was caused by the lack of periodic rest and sleep. Don’t forget to drink sufficient water and blink.
• Incorrect diagnosis: When eye strain gets bad, the way it affects the eyes can result in doctors assuming you have astigmatism or that your eye power is higher than what it actually is. Strained eyes can result in a power that is -0.25 to even -1.0 higher than what the eye power actually is. My eye power is -4.25, but at the worst stage of strain, after a full day of my office work, the doctor measured it at -5.5. Other doctors even prescribed cylindrical correction which I couldn’t tolerate for more than 10 minutes. Even computerised eye tests and pupil dilation tests showed astigmatism and the wrong power. Oddly, every doctor gave me a slightly different eye power prescription, and each time, the cylindrical correction angle was completely different from the previous reading. I started spending most of my day either not wearing spectacles or using spectacles of a lower power (which didn’t cause any issue, by the way), and when I started getting better sleep and rest, the strain reduced. This time, after getting a good night’s sleep, when I went to the doctor with well-rested eyes, my eye power was diagnosed correctly. Two doctors concluded I don’t have astigmatism.
• “Is this lens better or this one?”: It is not necessary to be able to clearly see the last line in the Snellen’s chart. As long as the spectacle lenses feel right, that’s the eye power to stop at. When my eye power was -4.25, and the doctor added another lens to make it -4.5 and asked whether the higher power was better, it appeared clearer and sharper to me (I didn’t realize it was too powerful), so I chose -4.5. I shouldn’t have done that. -4.25 was the correct power. When using the computer, I switch to -2.0 power ordinary glass lens spectacles without any coatings, and I’ve had absolutely no problem (spectacles purchased in 1998). Some people recommend wearing spectacles even if it feels uncomfortable or if it gives a headache, saying that the eyes would adjust after a few weeks. That’s very wrong advice. When my eyes felt uncomfortable, I learned not to wear those spectacles. They were the wrong power, and caused a lot more strain.
• Low power prescriptions: I’m hearing of a lot of people being prescribed -0.25 or -0.5 prescriptions for one eye or both eyes. Please consider that these powers may just be a result of excess strain or lack of sleep. Wearing spectacles could prevent them from recovering to normal sight.
• Bad habits from childhood: For goodness sake, please don’t mess up children’s sleep cycle. Not even for exams. Hard work is good, but losing sleep repeatedly to do hard work is a foolish thing to do. Right from childhood, we have been taught to ignore discomfort and persevere, but what we should have also been taught, is to recognise when we should stop and give our bodies the rest it needs.
• Online classes and smartphone/TV use: Teachers and parents need to ensure that students get some necessary breaks during online classes, smartphone use or while watching TV. Kids by nature won’t listen, but please ensure they get their rest, sleep and nutrition. Their disappointment at not being able to play video-games for long, is tiny compared to the disappointment of having to wear spectacles lifelong. The need to wear spectacles itself shows serious damage to the eyes. Don’t make it worse. Better still, don’t even allow it to reach a stage where anyone needs spectacles. Protect the eyes.
• Daytime sleep: If you couldn’t get proper sleep at night, you will feel sleepy during daytime. It is extremely important to take that nap. It gives some very essential rest for the eyes that’s even more effective than just keeping the eyes closed.
• Coffee/tea: When the body badly needs rest and you feel sleepy, you should sleep. Drinking coffee or tea to remain awake is a silly thing to do.
  
• The angle: If you extend your hand in front of you and hold a dumbbell for long, your muscles will start aching after a while. When it starts aching, if you raise your hand to hold the dumbbell above your head, some of your arm muscles will be spared of the weight of the dumbell and get a chance to relax. The extraocular muscles holding the eyeball, are like the arms holding the dumbbell. Closing your eyes and lying down facing the ceiling helps those muscles relax a little bit.
  
• Spectacles are not a cure: Some people assume that if they constantly wear spectacles or contact lenses for a few years, they will get cured to normal sight. It won’t.
• Glass lenses were better: I find this strange myself, but plastic lenses always caused my eyes to get strained quicker than when I used glass lenses. Now that I’ve recovered a lot, I feel that difference a slight bit lesser, but there seems to be some reality to it. Perhaps glass offers better visual acuity. I hope researchers could look into this. Of course there’s a safety benefit to plastic lenses, but while sitting in front of a computer, I prefer glass lenses. Some opticians may lie, saying that glass lenses are not available. They are available and are cheaper than plastic lenses.

As for everyone who knowingly or un-knowingly deprive people of sleep and rest right from childhood, please realise the seriousness of the matter and establish processes that help create a nation of healthy people.

The human eye was not evolved to handle this kind of strain. Too many people are getting incorrect advice and hurting their eyes.

Like someone once said: “If you do not give time for wellness, you will be forced to make time for illness”.

 

 

Keywords: computer eye strain, eye strain glasses, eye strain headache, reduce eye strain, eye strain causes, digital eye strain, eye strain from computer, glasses for eye strain, eye strain relief, eye strain in one eye, what causes eye strain, eye muscle strain, how long does eye strain last, why do my eyes hurt, eye drops for eye strain, glasses for computer eye strain, how to get rid of eye strain, burning eyes, eye drops for burning eyes, pain in eye, pain behind eye, right eye pain, left eye pain, headache eye pain, sharp eye pain, eye pain causes, sharp pain behind right eye, pain relieving eye drops, home remedies for eye pain, eyes hurt, pain above right eye, pain in eyeball, pain over left eye, how to relieve eye pain, how to stop eye pain, reason for eye pain, medicine for eye pain, eye pain home remedies, eye strain symptoms, headache and eye pain, eye socket pain, stabbing pain in eye.

Install PyEnv if you want various Python versions on Ubuntu

One of the major issues with installing new versions of Python is that if you try removing it or try removing the existing version, it'll cause problems with the working of the operating system, because Ubuntu uses Python for many of its system functions.

As an alternative, people suggest using Anaconda or Miniconda. When I tried it, I ran into errors where the package versions installed with Anaconda conflicted with the system Python versions. At my wit's end, I finally gave PyEnv a try and it worked beautifully.

1. sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl
2. curl https://pyenv.run | bash
3. Follow the instructions shown on the terminal to add commands to bashrc. Close terminal. Open a new one.
4. pyenv install --list | grep " 3\.[678]"
5. pyenv install -v 3.8.3
6. pyenv global 3.8.3
7. To see which versions of pyEnv are installed: ls ~/.pyenv/versions/ or pyenv versions
8. To remove any version: pyenv uninstall 2.7.15 or rm -rf ~/.pyenv/versions/2.7.15
9. Check which is the system default version: python -V
10. To revert back to the default system python: pyenv global system
11. Commands available with pyenv: pyenv commands
12. Getting help for any of those commands: pyenv shims --help
13. Check if pip is installed: pyenv which pip
14. Set a global version: pyenv global 3.6.8
15. Set a local version for a specific application's directory: pyenv local 2.7.15

The most impressive parts of PyEnv are the ability to switch between Python versions and have separate local and global versions.
Although the next step would be to install a virtual environment using pipEnv, I didn't like it because the updates portion is poorly implemented.

Keyboard cleaning

The good thing about freelancing is that one gets a good amount of spare time to relax the eyes and go for a digital detox. Add to that a global pandemic, and you get to learn some new things.
I decided to open my keyboard which had been gathering dirt for many years. Here's the "before" picture.

 

Opening it up showed how much dirt had accumulated over the years. But I also noticed that the reason that the keys didn't have a smooth movement when pressed down was not the dirt. It was because they had to slide down the plastic, and there was no lubrication.

 

This is the membrane the keys push down. It's a "dome switch" which completes the circuit when the finger presses it down.

The circuitry under the domes. I was surprised at how minimalist it is.

 

A word of caution. When removing the key caps, don't use a compass or an object that exerts force at a single point, because it'll cut a groove in the plastic, as shown in the image below. It's better to use an old debit card or a professional key cap remover or even make one yourself using metal wires.

After cleaning and drying, the keyboard looks good as new!!! I wanted to apply some grease too, but some forums and comments warned against using grease or oil without first checking how it'd affect the plastic.

Don't forget to take a photo of the keyboard before opening it up. It'll help when you are putting back the key-caps.
Next time I'd go for either an ergonomic mechanical keyboard or a buckling-spring keyboard.

When using the Hemmingway Editor, take the grading with a pinch of salt

For writing assignments, many people recommend the free Hemmingway editor, which help remove unhelpful adverbs, passive voice and in general, helps improve your sentences by making them simpler and easier to read. These features are really useful.

However, recently when I wrote some simple sentences, the editor said they were at Grade 11, when these were simple sentences that even a 7th or 8th grader could understand. So I did a little experiment. I chose sentences from two children's books and checked to see what Hemmingway editor would report.

First book: The call of the wild.

This is a version simplified for children, meant for 3rd, 4th or 5th graders. While most paragraphs showed a “Grade 5” or “Grade 6”, a sentence whose only fault was that it was too long, showed a “Grade 7” difficulty.

Moreover, copying and pasting the same sentence repeatedly, brought the score up to Grade 11. From there, it was not difficult taking it up to Grade 13. This is a serious flaw in the algorithm.

Second book: Chips (story of a rabbit).

This book has very simple sentences meant for children in the 2nd or 3rd grade. While most paragraphs showed grades ranging from “Grade 5” to “Grade 9”, some sentences gave these results:

Oddly, removing a few sentences brought it to Grade 10.

Removing more sentences pushed it to Grade 11.

By repeating the same sentence, I could push it to Grade 14.

I've written to the creators of the editor, showing them these examples. Hope they'd correct it soon. Meanwhile, writers should take the gradation score with a pinch of salt. The editor is still a good method of evaluating your text for the other features it provides.

Repairing grub in a dual boot system without using Boot Repair

After installing Windows, installing Ubuntu detects Windows and installs fine, giving you a dual boot system. But if you re-install Windows or install another OS on another partition, chances are you'll get a MokManager error that looks like this:

Before attempting to repair Grub2, go to your BIOS (as shown in this page) and select the grubx64.efi file or whichever efi file is relevant to your installation, and reboot. Chances are that the non-booting is just being caused because the computer does not know which efi file is safe. I think this happens due to secure boot being enabled, but please don't disable secure boot. Just specifying the correct efi file may help.

If it still isn't booting and you want to repair Grub2, follow the steps in this or this website. I had tried using Boot Repair, which worked nicely, but only once. So I now prefer doing it manually.

Steps:
1. Boot from a live CD or live USB which has the same Ubuntu version as your latest installed Ubuntu.
2. Use sudo fdisk -l and sudo blkid or GParted to determine which sda Ubuntu is installed on. Let's assume it's on sda4.
3. sudo mount /dev/sda4 /mnt
4. for i in /sys /proc /run /dev; do sudo mount --bind "$i" "/mnt$i"; done
5. If Ubuntu is on an efi partition (check using sudo fdisk -l | grep -i efi), mount it with sudo mount /dev/sda4 /mnt/boot/efi.
6. This command won't work if you are using a 32 bit live boot disk on a 64 bit computer, so make sure you use a 64 bit live boot disk: sudo chroot /mnt
7. update-grub (if this doesn't work, goto step 8. Else, step 10).
8. grub-install /dev/sda
9. update-grub. To locate and add Windows to grub menu.
10. If Ubuntu was installed in efi mode, run this: blkid | grep -i efi; grep -i efi /etc/fstab
11. Reboot the computer.

Even now, the "Failed to start MokManager" error may show up. Just go to BIOS and select the right efi file and reboot. Everything should work fine now.

Removing malware from PDF's

Sometimes PDF files can contain malware. Although the techniques listed below are not a fool-proof guarantee of getting rid of malware, it can get rid of malware that's embedded in the metadata of PDF files.

First, confirm that the PDF has malware.

Now create the following bash script in a file named malwareRemover.sh:
(make sure you use a tab instead of spaces for the lines within the for loop).

#!/bin/bash
clear
for f in *.pdf; do
    echo "Removing malware from $f:"
    echo "-------------------------------"
    echo ""
    #---convert
    tempFilename="temp.ps"
    echo "Converting $f to $tempFilename"
    pdf2ps $f $tempFilename
    echo "Deleting $f"
    rm $f
    echo "Converting $tempFilename back to $f"
    ps2pdf $tempFilename $f
    echo "Deleting the ps file"
    rm $tempFilename
done

Change the permissions of the file and install ghostscript:
chmod +x malwareRemover.sh
sudo apt-get install ghostscript

Run it with all the PDF files in the same folder as malwareRemover.sh.
./malwareRemover.sh


If you are using Didier Stevens' tools, this script will help:

#!/bin/bash
clear
for f in *.pdf; do  
    echo "Scanning $f for malicious data:"
    echo "-------------------------------"
    echo ""
    #---scan
    python3 pdfid.py $f
done

echo ""
echo "How to interpret the info:"
echo "------------------------------"
echo "* Almost every PDF documents will contain the first 7 words (obj through startxref), and to a lesser extent stream and endstream. I’ve found a couple of PDF documents without xref or trailer, but these are rare (BTW, this is not an indication of a malicious PDF document)."
echo "* /Page gives an indication of the number of pages in the PDF document. Most malicious PDF document have only one page."
echo "* /Encrypt indicates that the PDF document has DRM or needs a password to be read."
echo "* /ObjStm counts the number of object streams. An object stream is a stream object that can contain other objects, and can therefor be used to obfuscate objects (by using different filters)."
echo "* /JS and /JavaScript indicate that the PDF document contains JavaScript. Almost all malicious PDF documents that I’ve found in the wild contain JavaScript (to exploit a JavaScript vulnerability and/or to execute a heap spray). Of course, you can also find JavaScript in PDF documents without malicious intend."
echo "* /AA and /OpenAction indicate an automatic action to be performed when the page/document is viewed. All malicious PDF documents with JavaScript I’ve seen in the wild had an automatic action to launch the JavaScript without user interaction."
echo "* The combination of automatic action  and JavaScript makes a PDF document very suspicious."
echo "* /JBIG2Decode indicates if the PDF document uses JBIG2 compression. This is not necessarily and indication of a malicious PDF document, but requires further investigation."
echo "* /RichMedia is for embedded Flash."
echo "* /Launch counts launch actions."
echo "* /XFA is for XML Forms Architecture."
echo "* A number that appears between parentheses after the counter represents the number of obfuscated occurrences. For example, /JBIG2Decode 1(1) tells you that the PDF document contains the name /JBIG2Decode and that it was obfuscated (using hexcodes, e.g. /JBIG#32Decode)."
echo "* BTW, all the counters can be skewed if the PDF document is saved with incremental updates."
echo ""
 

Which conferences and journals allow pre prints on ArXiv and which don't?

ArXiv, TechRxiv (owned by IEEE) and HAL are pre-print repositories where scientific papers can be submitted as proof of precedence, so that the ideas do not get scooped. Especially when you are submitting a paper to a journal, it's better to put it on a pre-print repository, because the journal's review process can take very long and the process of making corrections and resubmitting can take many years until the paper finally gets published.
But there may be conferences or journals which may not allow authors to publish their work on pre-print repositories, because it would interfere with the double-blind review process.


These are conferences/journals which allow authors to publish on ArXiv:

• IEEE allows it. They have some formal policies about it.
• Elsevier allows it. 
• Springer allows it.
  

These are conferences/journals that don't allow it:

• KDD. Prior to 2019, KDD used to allow pre-prints on ArXiv.

I hope to update this page with more names of conferences and journals. If you know of any, please mention it in the comments.

GMail's autocorrect messing up your email?

We don't allow children to drive cars, but somehow people think it's ok to allow a self-driving car on the road...an algorithm that has no clue about what the world is and no clue about what driving is. AI has a long way to go before becoming intelligent.

Similarly, GMail tended to autocorrect our words for us, without even highlighting the words it autocorrected. The result? I typed "presenter" in a sentence and in the next sentence when I typed "presenting", GMail discreetly autocorrected it to "presenter", giving the sentence a grammatical error which gave a bad impression to the reader. I noticed the error only after sending the email. The minimum GMail could have done was to highlight the word that had been autocorrected or to ask for my consent before introducing the feature. Google seems to need better engineers.

When I first heard that a person lost his job because of autocorrect correcting a word to "hash brown" when he emailed a colleague, I didn't actually believe it happened unintentionally. But after seeing how autocorrect messed up my email, that man has all my sympathies.

There are many such examples. Some of these may be fictional, but they do drive home the point.

Autocorrect can be switched off in GMail in the General Settings. I recommend switching off all of the AI features that aren't reliable.

IISc Open Day 29th Feb 2020

The Indian Institute of Science's Open Day is a yearly event during which the general public can visit the campus free of cost and every department exhibits some fascinating experiments or presents the research that students are doing there. Most of these are at the cutting edge of technology, so it's an excellent learning experience for everyone. The campus is so vast that even for someone arriving at 9am, it's hard to visit all the stalls and view all demo's and experiments. Of the many demo's, here are some:

Also known as Magic Mud or Oobleck, the video I took, shows how a person who inserts their hand slowly into the liquid is able to, but the person who jumps on it does not sink. The liquid is just a 1:1.25 ratio mixture of water and corn-starch. The corn-starch doesn't dissolve in the water, and stays suspended as a non-Newtonian fluid. If you are gentle, the liquid parts and allows solids to enter. If you punch it or run over it, if feels like a solid object. I took a thin, sturdy leaf stalk and thrust it into the liquid, but it wasn't able to penetrate. So surface area didn't matter. I'd like to have seen if a knife edge's surface area would be able to penetrate. When I suggested it could be used for bullet-proofing, the organisers said that's one of the applications considered for this fluid. I jumped on it, ran on it and even punched it. Solid as a rock! Be gentle, and it's cooperative. Made me realise that many people behave like non-Newtonian fluids too. :-)

Studies are being conducted on the structure and role of certain proteins which play a role in how ambient light affects the body's sleep cycle.

This was an actual brain-machine interface demo where the person wearing a skull-cap that noted his brain's gamma waves, was able to control a circle on the screen and make it move left or right just by using his thoughts.

Hardware relics from the past.

The presence of such ponds and greenery in an institute campus speaks a lot about the quality and care that goes into maintaining it.

This is a pacman game where they train a deep neural network using the visitor's hand gestures that indicate up, down, left or right motions. The visitor then uses these gestures to move pacman on the screen and play the game of escaping the ghosts. Cumbersome, but interesting.

A functional hydroponics setup that's 90% more water-efficient than traditional farming. The nutrient-enhanced water mist gives nutrition to the roots.

Just a car that follows a trail using infra-red light.

Research being done on mice to understand how the hippocampus stores information on spatial recognition. The brain has grid cells, head direction cells, boundary detection cells etc.

A working demo of the Tesla tower. The CFL bulb glows due to wireless electricity (ionization).

A demonstration of how Tsunami waves are formed, how the wave appears to recede just before the main wave strikes and how barriers may help push the wave backward to cancel out incoming waves; demonstrated by a guy who appears to be infected with the Corona virus :-).

One of the e-rickshaws at IISc that the public could use for free. They are labelled "Transvahan", which initially gets interpreted as "Transgender vahan", but it's actually "Transport vahan". There's very few of these vehicles, and most of the time I saw them transporting water and food. It would've been nice if there would be plenty of bicycles that'd help traverse the vast expanse of the campus during such events. They did a really good job of arranging for plenty of good quality drinking water sources all over the campus.

There were so many many more stalls and demo's that I didn't mention just for the sake of brevity. Overall, a really good experience. Open Day is a must-visit for science-lovers.

Configuring LiClipse to run Anaconda or Miniconda

I'll have to admit, it is indeed better to use a virtual environment when dealing with Python. If you mess up your native Python, the system won't work properly. But the problem with using Python this way, is that there may be packages that aren't available for install via conda for particular versions of Python. That gets very annoying.
This article explains how to setup LiClipse to work with Anaconda or Miniconda. Specifically, Miniconda.

Open LiClipse. In the main menu, goto Window > Preferences.
Type "interpreters" in the search box.
Click "Python interpreter".
Click the "Browse for Python/pypy exe" button.
Close the first window that opens and you'll see this:

Browse to your Miniconda folder and select the python3 or python file in ~/miniconda3/bin.

In Interpreter Name, enter a name of your choice: MinicondaPython3.7, for example.

One more window opens up. Click Ok.

Now (assuming your Python project is open), in the main menu of LiClipse, select Run > Run Configurations.
In the left column, right click on Python Run. Select "New Configuration".

Under the "main" tab, choose your project name and specify the main file.
In the "interpreter" tab, select MinicondaPython3.7.

That's it. Install any necessary Python packages using the conda command, and you are ready to run your project!

Clearing swap space in Ubuntu

When having two large applications open in Ubuntu can cause slow performance because of swapping memory from hard disk to RAM, it leaves one perplexed to see swap space still being used after all applications are closed.

I investigated this a bit more to find out that it's better not to mess around with the kernel's swappiness parameter, and that using the "top" program or other scripts is not really necessary. You don't even need to use vmstat to check the si and so columns to see if swaps-in and swaps-out are happening.

There's a far simpler option that promptly copies all swap memory back to RAM.

sudo swapoff -a; sudo swapon -a

I opened a large application by the time I took the screenshot, hence the greater usage

Enjoy!

Converting webp files to more familiar formats in Ubuntu

With Google's efforts at better image compression, came an acquisition that gave rise to the webp image format. Although it was announced in 2010, I was surprised that Ubuntu 16.04 didn't support it natively.
To use it with Gimp, you need one of the latest versions. Good news is, ImageMagik supports it if you install webp. Once installed, you can batch convert your webp files to png too. Two simple steps.

To install:
sudo apt-get install webp

To convert all webp files in a directory to png:
find ./ -name "*.webp" -exec dwebp {} -o {}.png \;

To delete the original webp files after converting to png (though I'm sure there has to be a better way):
find ./ -name "*.webp" -exec dwebp {} -o {}.png \;; find . -type f -name '*.webp' -delete

The precious thing frequently stolen from you: Sleep

More than seven years of noting observations and trying to correlate the causes of severe eye strain and poor vision, led me to the root cause: The lack of uninterrupted deep sleep for seven to eight hours every night. The funny part is, that if you ask anyone if they are getting enough sleep, they'll immediately say "yes", even though they are actually getting only around four or six hours of bad sleep. Even I said "Yes" to the first doctor who asked if I was getting sufficient sleep (when my eye strain was at its severest and I was getting only 4 hours sleep).

There are bodily repair processes that do not get completed if deep sleep is interrupted before 8 hours, and the debt accumulates over the years. I've confirmed this multiple times. There's a huge difference between "resting" and "8 hours uninterrupted sleep". I have good reason to say it's the lack of sleep and lack of proper nutrition that end up giving people weak vision and a lot of other health issues. Six hours or four hours sleep is not enough. The people who have been insisting for all these decades about seven to eight hours, were always correct.

Did these affect you?

• Children being woken up early for school.
• Children being told to stay awake late or wake up early to study during exams.
• Children staying up late to watch TV or play video games etc.
• Adults being forced to work late due to deadlines.
• Having to use an alarm clock to wake and not be late for work or to cook breakfast.
• Doctors being woken up for emergency cases.
• Family members of patients being woken up during a hospital stay.
• Taxi drivers and auto-rickshaw drivers losing sleep to make ends meet.
• Phone calls and extraneous noises when sleeping.
• Health issues that cause people to wake up once or more at night.
• Improperly cooked food or adulterants or burnt particles in food affecting the digestive system and causing sleep loss.

Sleep Requirements researched by the National Sleep Foundation.

Is there anything we can do to change the culture of sleep loss?

• Insist on properly cooked, unadulterated, un-burnt, healthy food and water at home, canteens and restaurants.
• Allow for flexible work cultures and work-from-home options.
• Allow schools, colleges and offices to begin at 11am for some people (night-birds) and at 9am for other people (larks). 
• Spread awareness of this at school level and get a buy-in from parents and employers.
• Arrange for better work shifts and recruit more people to handle workload.

We need to create ways to make this happen in a cut-throat rat-race. We've been through centuries of brainwashing about the heroics of equating hard-work with not sleeping. "How many people's health are you permanently willing to ruin in order to make a living?". Losing sleep even occasionally is not good. Don't allow people to nudge you into it. Don't nudge children into it.

There's no heroism in losing sleep.

Healing eye strain: https://nrecursions.blogspot.com/2018/03/eye-strain-is-not-caused-by-digital.html

Copy pasting and clearing screen in the Ubuntu terminal

Copy pasting with Ctrl+c and Ctrl+v works fine on the MacOS terminal, but doesn't work in Ubuntu. I assumed this may be some security feature or because Ctrl+c is the age-old command for an interrupt, so I didn't give it much thought and just used the mouse to copy paste.

But today it just struck me that I should find out the real reason, and viola! It's just a matter of pressing Shift.

To copy:
Shift+Ctrl+c

To paste:
Shift+Ctrl+v

and as a bonus:
To clear screen on the terminal:
Ctrl+L

This clear screen also supposedly works in Matlab, MySQL etc., though I can confirm it works on the Python terminal.

Aquaplaning, and how to avoid it

Long back, I was curious about why Formula One racing cars had such "smooth" tyres. The racing slicks. Turns out, when there's no water on the road, it's actually more beneficial for more of the tyre to make contact with the road and provide more grip. Having grooves would lessen the grip. When the road is wet however, the grooves/treads on the tyres are absolutely necessary to prevent aquaplaning. The rain tyres allow water on the road to enter the grooves, thus allowing the remaining part of the tyre to make contact with the road and give better grip. Without this, the car would go completely out of control.

Exactly the same principle is used when designing footwear. The process of cutting such slits is called Siping. Recently, I purchased a new pair of rubber slippers and noticed that it didn't give any grip on wet surfaces. I even slipped and fell down once.

On examining the sole, I realized why.

No grooves.

So I took a knife and cut out some crude grooves all along the front, mid and back portions of the slippers. Just these crude grooves were enough to give a good grip on wet surfaces. I also wrote to the company, chiding them for not considering this basic design element. They wrote back thanking me for the valuable feedback and stating that they were the number one footwear brand in the country. The number one brand? With a safety aspect like this disregarded, I definitely don't consider them number one, but I'd be willing to consider it if they make the necessary design changes.

Update: The horizontal groove I made was a mistake. When the slippers flex when walking, the groove deepens and causes the rubber to break at that point. So the better option is to either cut grooves vertically or just buy new slippers with better soles.

Realtime plotting in Python

If you intend to create plots/graphs with Python, your first choice may be Matplotlib. Don't use it unless your plots are extremely simple and isn't expected to grow more demanding with time. I used Matplotlib for realtime graph plotting for a really tiny graph and it was disappointingly slow. Moreover, it required installation of Python 3.6, which wasn't available readily for Ubuntu 16, and caused even more problems.

Then I came across PyQtGraph, and was impressed by the super-fast plotting speed and the mind-blowingly impressive variety of graphing functionalities. Really. You gotta see this.

Install it using: pip install pyqtgraph
Run the examples with: python3 -m pyqtgraph.examples
And prepare to be impressed!!!

I've come to understand that the rendering is quick because the underlying code is in C/C++ and the Python bindings are merely meant for invoking that code.


Still, even though the graphs are very impressive, the non-linearity of the code examples and the lack of code comments make it really hard for a newbie to understand how to get a basic graph running.

Here's where I help. After spending an extraordinarily long time figuring it out on my own, I've created a simple, basic example with which you can create a realtime graph. Hope it helps...


import time
import random
import pyqtgraph as pg
from collections import deque
from pyqtgraph.Qt import QtGui, QtCore

class Graph:
    def __init__(self, ):
        self.dat = deque()
        self.maxLen = 50#max number of data points to show on graph
        self.app = QtGui.QApplication([])
        self.win = pg.GraphicsWindow()
       
        self.p1 = self.win.addPlot(colspan=2)
        self.win.nextRow()
        self.p2 = self.win.addPlot(colspan=2)
        self.win.nextRow()
        self.p3 = self.win.addPlot(colspan=2)
       
        self.curve1 = self.p1.plot()
        self.curve2 = self.p2.plot()
        self.curve3 = self.p3.plot()
       
        graphUpdateSpeedMs = 50
        timer = QtCore.QTimer()#to create a thread that calls a function at intervals
        timer.timeout.connect(self.update)#the update function keeps getting called at intervals
        timer.start(graphUpdateSpeedMs)   
        QtGui.QApplication.instance().exec_()
       
    def update(self):
        if len(self.dat) > self.maxLen:
            self.dat.popleft() #remove oldest
        self.dat.append(random.randint(0,100)); 

        self.curve1.setData(self.dat)
        self.curve2.setData(self.dat)
        self.curve3.setData(self.dat)
        self.app.processEvents()  
       

if __name__ == '__main__':
    g = Graph()
   
   

Unable to signout / logout of GMail?

The fact that GMail used via the mobile web browser has such a silly bug that didn't get fixed even after I reported it thrice, is a sad state of affairs.

Anyway, if the signout menu isn't appearing, simply type "google.com" in the address bar and you'll see a signout button at the top right corner.

That's it!