Shobhit Kumar

Linux & Opensource - Should we tame the beast ?

Tue, 27 Nov 2018 00:00:00 +0000

After been working in Linux for so long, I thought none in today's technology oriented market would be untouched by Linux in someway or the other. With companies embracing open soure technologies with wide open arms and making huge investments in open source development Linux has become the poster child for all tech companies.

Interesting read: Celebrating 25 Years of Linux Kernel Development]

So it was a surprise for me when I recently visited one of the prestigeous collge for a Linux workshop and found that Linux usage is negligible or very limited. Brief interaction with the students confirmed my reasons for the same. Today all technologies we have especially cloud oreinted are cross platform and easily work on other systems like Windows or MAC OSX. Then what is the need for Linux. I thought it was time to pen down my thoughts on why we need to learn Linux and what is its importance in todays world. Let me just start by giving some data on what kind of companies are focussing on Linux and Opensource in general -

Source: LinuxFoundation Kernel Development Report

Be it companies producing low power sensor devices to complex SoCs or Web services, everybody is hooked to using open source tech. Its not only the cost advantage that everybody sees. After all you need engineers to enhance, maintain and bring out best in your chips. So then why embrace Linux ? Its the quick time to market and quality that attracts tech companies. Just to give an example of Linux kernel on how Linux kernel has grown over the years, refer to figure below -

And following fugure gives details about all the contribution for one kernel version 4.13 -

Source: LinuxFoundation Kernel Development Report

So why more and more companies are moving towards OpenSource and Linux ?

The main reasons can be listed as below -

Highly Flexible and Scalable
- Being used in small smartwatches to huge media transcode servers today
- Fit for simple home usage to complex and huge Data centers
- Highly scalable distributed computing architecture and software availability
- Freedom to customize as per your business needs
Very Portable
- Works on ARM, MIPS, Intel, AMD and varied kind of hardware architecture
- Can achieve small footprint for real resource scarce environment
Secure, Reliable and Fast
- Open source philosophy provides a mature software tested by thousands of engineers and companies
- Time to market is very fast
High Quality Code and Design,
- The only project where Tech Giants working together and not against each other
- Highly scalable and robust design is the common goal
Longevity and Maintainability
- Continuously maintained by a community of people not a company
- Be seeders and the community will maintain your software
- Newer upgrades bring in all your changes along with improved features and bug fixes
- Freedom from Vendor Lock in
And of course its Free

Today open source and Linux has become so pervasive that companies working in Big Data, Cloud Computing, Databases, Machine Learning, etc. all use one or the other variant of Linux OS and rely heavily on the Development Tools from the Linux/Open Source eco-system. All open source tools are developed for Linux first and then made available on other platforms. For example Apache Web Server which drives more that 70% of the web servers in the market was first developed for Linux. And there is no doubt that we should always develop in the same environment as where we would deploy our production solutions. Further to underline the importnace of Linux and Open Source in the industry, if we look at how companies are engaging with open source we find that -

65 % of the companies are contributing to Open Source
67% of the companies encourage developers to participate
Only 1 in 3 companies have full time dedicated resource
- Lack of skills to participate in open source projects
- Not yet truly understood that they need to be seeders than lechers to reap the benefit of Open source
67 % participate to fix bugs or enhance
59 % do it to get competitive advantage
34 % of companies have 50% of their developers contributing to 1 or more open source project
50% of the companies have no formal policy for selecting and approving open source code
One of the tool - GIT is the overwhelming choice for SCM - 73 %

From students perspective if we look at what managers are looking while hiring, we find -

If we look at what motivates the professional developers out there in the market we find -

So to answer the question if we should learn Linux and Open Source devlopment tools, technologies and environment, I can give a very emphatic !!YES!!. It is absolutely important for anybody in Software/IT career to focus on Linux. My recommendation to engineering students is to make Linux as their primary Learning and Development tool, because its everywhere and is so versatile that it can be used in mutiple learning contexts. There is a rich community of experts and beginners alike which forms an excellent learning platform.

Industry is looking to hire Linux talent and honing ones' skills at college level is the best way to be job ready. Running Linux and working in Linux and solving problems in Linux, you will not only learn computer and software architecture but problem solving skills that are invaluable in ones' career.

So I would encourage every body who had read this far to embark on their journey of learning and take it up as a challenge which you are willing to accept and unwilling to postpone and determined to win.

Note:
1. All references has been taken from The Linux Foundation reports on Kernel Dev and Job Market in Open Source 2017
2. This is first part of the series on exlaining why we need to dive into Linux. Please stay tuned for upcomming short tutorials on setting up and starting learning on Linux. Please put in your interest in comments below and topics you would like to learn.

Create Customized Linux Distro

Mon, 16 Jul 2012 10:16:17 +0000

I own a HP Mini210 with Intel Pineview processor and all of the latest and greatest Linux distro sucks on this one. And those Win guys who may be reading this, don’t ever mention to me on trying Win 7 on this one, though it came with home basic by default, that goes against my morals to have such a thing like Windows on my machine less even talk about it.

So here I am on creating my own distribution customized to my taste and perfect the way I want it.

Few questions I asked my self about what I want to do once I am done with this distribution and thats what I concluded -

Boot as fast as it can
Able to browse internet
Download my mails
Watch on-line/local stored movies. HD content possible ?

So the above needed only simple packages, but I wanted something more - An easy to use package management system and I zeroed on Debian tool “aptitude” and my base distro as Debian. So lets jump into creating it right away.

What you need to achieve this

Any Linux desktop environment - Fedora/Ubuntu/Mint/Suse/Mandrake, etc
I see pain on Windows guys :) So install
- VirtualBox
- And then install any of the above mentioned flavors of Linux distros inside that
Do install debootstrap, make, gcc

Creating the base root file system

Now you can directly install this on a physical hard disk as well, but I will give steps based on virtual disks so that we can burn them as Live CD, you know the cool stuff to boot from USB pen-drive any-time, anywhere and yes persistent.

Create a virtual disk -

dd if=/dev/zero of=rootfs.ext4 bs=1024 count=524288

Yeah 512 MB is more than sufficient. We can resize it later. Keeping it small helps in actually while burning on USB pen drive. Actually the base console install needs just 256 MB. Blown Away Eh :)

Prepare the disk image file with ext4 file system and mount it. Next create the basic root fs.

mkfs.ext4 rootfs.ext4
mkdir rootfs
mount -o loop rootfs.ext4 rootfs
debootstrap --arch ia64 squeeze rootfs http://ftp.us.debian.org/debian

You have the rootfs ready and now time to use another cool thing - chroot

mount -t proc proc rootfs/proc
mount -t sysfs sys rootfs/sys
mount --bind /dev rootfs/dev
chroot rootfs /bin/bash

You are now in your root file system and any modification you do are restricted to your new debian root fs. Do configure the networking stuff, like modifying the /etc/hosts, /etc/network/interfaces, /etc/resolv.conf, /etc/hostname, etc

Also remember to add a new user for your self using useradd and also change the root password using passwd command. In Fedora selinux might crip, so disable that. Now our rootfs is ready. Just unmount everything

exit
umount rootfs/proc
umount rootfs/sys
umount rootfs/dev
umount rootfs

Building the Kernel

I wanted to use the latest Linux kernel out there and grabbed the tarball from www.kernel.org for latest stable 3.4.4 kernel

Apply the default kernel configuration x86_64_defconfig Now here comes the difficult and iterative task of fine tuning your kernel to support your device and only your device :) I removed all the unnecessary drivers. Just selected exact drivers matching my hardware for the HP Mini 210. Sometimes things didn’t work and you relise you have removed wrong driver and need to rebring it in. Now as I wanted to make it boot fast. Most of the drivers for the netbook I just made as kernel modules to have a very small kernel. Actually basic kernel booted in less than 3 seconds.

Putting things together with a boot-loader (Grub)

Insert you pen drive and make two partitions using fdisk. Note the device name carefully - sdb or sdc, etc

boot partition - 64 MB is more than sufficient. Format as vfat(mkfs.vfat)
rootfs partition - 8 GB for now. You can resize it later
Make additional partitions as you wish. Basically the current system to be usable you may need nothing more than 1.5 GB

Mount the vfat partition say as /media/boot. Install the grub as follows -

grub2-install --force --no-floppy --root-directory=/media/boot /dev/sdb1

Copy the default grub.cfg from your host linux system /boot/grub2/grub.cfg /mnt/boot/boot/grub2/grub.cfg. Remove all the enteries and just add a new entry for our latest kernel as follows -

menuentry 'TUX Kernel (3.4.4)' --class gnu-linux --class gnu --class os {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_msdos
	insmod ext2
	set root='(hd0,msdos1)'
	echo 'Loading Tux Kernel (3.4.4)'
	linux   /bzImage root=/dev/sdb2
}

Burn the rootfs, resize it to 8GB and label it

dd if=rootfs.ext4 of=/dev/sdb2
e2fsck /dev/sdb2
resize2fs /dev/sdb2
e2label /dev/sdb2 rootfs

Copy the kernel image to the /boot partition

cp arch/x86/boot/bzImage /media/boot

We are ready. just boot from the pen drive and you have console environment booted in under 8s. Install the X environment as follows -

apt-get update
apt-get install xfce4

Now we have a bare minimum Linux system (Under 1 GB). We can fine tune what packages we want like firefox(browser), thunderbird (e-mail client), vlc(media player) etc avoiding all the clutter that all of the latest distros install by default. I got my desired system in less than 1.5Gb booting in around 10s.