Shiny Blog

Oh, hi… Let me introduce myself. My name is Nicolas Crovatti, I am a self taught developer and easily distracted by shiny things. This site, is my very own Shiny Thing and distracts me from my other activities. A shiny thing in my opinion ranges in order of importance from anything to new programming languages (new to me at least). This site is my playground. I am learning a brand new programming language, Ruby and the Rails framework for instance.

Over the years I developed various personal websites. My first (Frontpage generated HTML junk does not count!) was a portal (buzzword from the previous millennium) named Genius-line.com; back in 1999, web hosting was not cheap especialy when you were interested in learning ASP v3.0 and had no spare cash. As a matter of fact, this was a huge sacrifice (not counting my time) in the name of what I was already calling my Passion.

Genius-line.com provided basic web stat analysis for friend’s websites using javascript and ASP to gather informations and log everything into a MS Access database. This application gave an idea to my my boss in a ShopBot Start up, and this tool evolved into a complete reporting/billing system to track leads to our customer’s products.

At the beginning, it was powered by ASP and MS Access, the database quickly grew up and we had to migrate the back end to SQL server, and then a few months later, to Oracle because of poor speed performance.

Then, one night in 2000, out of fucking nowhere, came PHP. Web hosting became cheaper, in fact, it became free at a huge price: uncontrolled advertising was automatically injected in your pages.

Back to PHP, I started to learn this language (v3.0 at that time) by creating a developer’s community called Gencoding.com. The editorial content and targeted audience where mainly french speaking people. Meanwhile I was also learning Perl, so I was hosting both PHP and Perldoc mirrors on my domain as part of my effort to contribute to these beloved languages. Gencoding.com’s community members were mainly friends and co-workers. The site itself, despite some nice features like online helpers who were registered members and developers available for a chat on ICQ (Most popular chat system at that time) to help others newbies, a nifty shoutbox and a karma system for posts, only lasted one single year because of domain renewal procrastination.

The Shiny thing had shattered.

In 2001, I totally ported the reporting system at my job to PHP and MySQL and was very excited to do so as PHP offered various new and therefore shiny extensions. Graphics in the old version of the reporting system was made of images tags organized by table cells.

 100%
 - --- ---
 | | | | | <-- Transparent gif
 | --- | |
 | |#| ---
 | |#| |#|
 | |#| |#| <-- Colored
 | |#| |#| 1px image
 | --- ---
 |________________________ _ _ _
 0 1 2 3 ... etc

One of these extensions was (and still is) GD. Coupled (or not) with JPGraph I made a step forward into graphic’s generation.

I just love graphics. They are colorful and therefore, Shiny.

In 2002, I managed to learn the dark arcana called Linux. I did not learned ALL the Linux you fool, but big enough to feel myself comfortable to host my own web site and services.

* Sudden flashback *

Over the past several years, I installed many linux distributions. My first try was a disaster. I totally lost my windows 95 install. That experience kept me away from linux for a few months. Finally in 95 or 96 I managed to successfully install a RedHat 4.0 distribution. I had to downgrade my graphic card in order to run AfterStep on X but no matter, IT WORKED. I could do *almost* everything I could already do with Windows but, almost was not enough to stick with it. You know, Games and Internet Explorer were essential applications that I couldn’t live without.

A Bernard Madoff’s failure sized numbers of linux installations later, Debian hooked on me. I guess Linux’s user interface improvements and installation process easing along with personal evolving and self documentation about linux’s and *nix systems potential and spare hardware were the major factors why I switched.

2002 was also the year of my Do It Yourself Shiny thing.

Counter-clan.net, the domain name says it all. A full featured CounterStrike community site with clan rankings, clan ladder, Game servers informations, and so on.

It never happened.

Not shiny anymore CounterStrike was. After several years of almost complete dedication to this game, the impulse to create the Counter-clan.net masterpiece failed me.

Nevermind Counter-clan.net became my homepage. Here is a quote from the welcome page :

Welcome to the Counter-Clan Network! Actually this site is a multi-purpose-web-site, I mean you may find not strictly Counter-Strike related content (I love this sentence). In other words, it’s a personnal homepage with a nice domain name. :)

Here is a link to web-archive.org : http://web.archive.org/web/20020604090613/http://www.counter-clan.net/

I created various interesting things hosted on my good old AMD Duron @800mhz! Here’s a list for my own record (Remember, you are visiting my shiny thing):

Counter Spy

Perl CGI that queried Counter Strike servers for various informations like players, scores or free slots. Note that the CGI was a port of a functionality of LazVegaz my IRC bot written in Perl that haunted efnet for years.

IMDb PHP Class

My first OpenSource project. An interface that permitted visitors to search the imdb site from your own site, display summary informations and casting about movies and saves results along with poster into mySQL for caching purpose. Fun with Binary Large Object field and regular expressions.

Fellow developpers found this project interesting and were involved in creating plug-ins and adding new features for various popular CMS like PostNuke, PHPNuke and Xoops.

As part of these developers, Greg Allan aka Adam Baum, co-founder and lead developer of PostNuke who died on June 16, 2002 in a motorcycle accident who ported IMDb PHP Class to PostNuke.

The project has been closed shortly after I received a notice from a dude at IMDB.com who was not aggree with me about advertisements stripped from content. Or something like that.

Webalizer patch

This patch was mainly an aestethical update to the popular Webalizer web stat analysis software. It mainly consisted of CSS support to skin Webalizer.

Various guides

Since CVS was the FOTY, as part of imdb php class, I proposed a cvs server for involved developers. This guide was a mini Quickstart Howto about using CVS.

I was very proud of my box uptime! I can’t remember exactly how much it was: web-archive.org states 98 days but I’m fairly sure it was a 3 numbers figure starting with 2 before the Teh Blackout.

Teh Blackout

In 2003, like I just stated, my server was running really well! I even was hosting my own smtp server while struggling with spam and Procmail. Then on summer 2003, I was on a trip in Berlin, Germany. Even if a one-week trip away from my long time running box seemed trivial, the Murphy’s Law stabbed me in the back.

I was checking once a day my sites stats and new forum’s posts from a cyber coffee near my hotel till the moment it couldn’t even respond to ping requests.

I knew later when back home what happened. Server’s aging motherboard and hard drives couldn’t handle the hit. My two years of personal work had been defeated by a single capricious thunderstorm. And mainly the fact that I never, even thought about backing-up anything, you know, all was so steady … Linux … Steady … Uptime …

Lesson learned.

[ insert random morpg addiction during 2004 ]

I guess I had to take a new breath.

In 2005, I tried to follow the blog way of life. My first blog was created that year, and was about me. I did not find enough to tell about this subject and stopped writing about a month after launch.

2006 was the year of new geographical location, new job, new projects to work on. Nothing I can expose here.

Over 2007 and 2008, I started moblur.org

Moblur.org was my second blog. I purchased a dedicated hosting for this one because I wanted to talk about what I know best: programming languages.

And I did well. I was writting about a subject and developed it in a “Workshop” with proof of concept and more in-depth explanations.

From these posts, an idea came to me, building a favicon archive. That has never been done yet; that was useless so I found it to be necessary. I started the crawler coding in PHP5 backed up by a Postgresql database.

I quickly organised various cron jobs to manage my crawler’s instances. Database and filesystem started to grow really big, so I had to ask for help to my fellow colleague Phil concerning PostgreSQL indexes strategies. And then, when all was running flawlessly, teh Shiny faded.

When I stoped the hosting contract for moblur.org the 250GB web partition was almost full and the database was 9GB big. I learned a lot about web crawling and large scale web applications from this experiment.

For instance, I had to find a smart way to store a huge number of little favicon files (some were not that little and easily broken the 10 megs line). I first saved them by using reverse fqdn here is an example for www.shinylittlething.com/favicon.ico :

com.shinylittlething.www.ico

That was cool, I could easily find all icons related to a specific domain name but it was flawed. What about millions of files in a single folder ? System Management became harder above 10k files since standard bash commands failed to deal with such large list.

I then recalled Sourceforge directory structure for users’ home organisation in their shell environment.

When I first noticed this, I was like: OMG, how dumb they are to create sub-directories named of single letters that occurred to be the first couple username letter?! /home/users/n/c/ncrovatti/ lol!

Several years later, I finally understood.

Huge amount of files are easier to manage when split in small chunks. I promptly created the bash script to create such directory structure and another to sort already existing files.

This directory structure gave me the opportunity to enhance my backup system while still allowing alike navigation.

In 2009, I’m now hitting the 10 years experience mark. I’m still really glad with the career that chooses me. I’m still passionate and learning a huge amount of new techniques and improvements everyday, the nature of technologies is to be a constant evolution. That hints me that I am still *far* to be bored.

So far my friend… Despite my bad english, you have read this self-centred article till the end; I just have to consider you as a friend for this mark of interest.

See you my friend, see you in another 10 years after 20 years of shiny distractions and way sooner in my next article “The Genesis of a developper”.

I couldn’t safely close this article without thanking zejulio for the invaluable help he gave me by fixing my english.

Trully yours,

Nicolas Crovatti.