normalize int data

Cool, Alan!

 

 

Your math skills are so cool!

 

 

regards,

George

Yes, 59 will be normalized to (approximately) 0.5. I always start by looking for the midpoint of the range to get an idea of how much skew the normalization is introducing.

If you consider the position when i == min, then temp is zero and the normalized value is:

  Exp(Log(Lower + 1)) - 1

which is:

  Lower + 1 - 1 = Lower

Similarly for i == max, then temp is unity and the normalized value is:

  Exp(Log(Lower + 1 + Higher + 1 - Lower - 1)) - 1

which reduces to:

  Exp(Log(Higher + 1)) - 1

which is:

  Higher + 1  - 1 = Higher

and it's now easy to see that any intermediate input will be normalized to a value in the interval [Lower, Higher].

Cool, Alan! You are making Finance as well as C#!!

 

Two more comments,

 

1. 0.5 corresponding to an integer of  59, does it mean input 59 will be normalized to 0.5?

2. how do you prove "return Math.Exp(Math.Log(Lower+1) + temp * (Math.Log(Higher+1) - Math.Log(Lower+1))) - 1;" will be in range [lower, higher]?

thanks in advance,
George

I'm not aware of any 'official' normalization methods though it's a long time since I did any serious mathematics and I would think, in any case, that it depends in which field you're working and the probability distribution of the variable you're dealing with.

I just did a search on logarithmic interpolation and came up with this link:

http://books.google.co.uk/books?id=tt3RLXsagagC&pg=PA74&lpg=PA74&dq=logarithmic+interpolation&source=web&ots=LfPy-51BHH&sig=X8R07MZRLPRyrDgtp9zNm2Yu4TU&hl=en

This suggests yet another approach:

    public double LogNormalize2(int i)
    {
       if (i <= Min) return Lower;
       if (i >= Max) return Higher;
       double temp = (double)(i - Min)/(Range - 1);
       return Math.Exp(Math.Log(Lower+1) + temp * (Math.Log(Higher+1) - Math.Log(Lower+1))) - 1;
    }

Once again I had to add one before taking logs to avoid NaN values though I then subtracted one to balance it out.

The results were pretty good with 0.5 corresponding to an integer of  59 so this time the results are skewed towards the lower half of the range, though not too much. This should be suitable for a lot of more distributions than the last approach.
 

Cool, Alan!

 

I am wondering whether there is any official logarithm or linear based normalization methods (e.g. MD5 has official implementation in RFC), or there are just some experience based or best practices based implementation like you did in this post?

 

 

regards,

George

The best way to test that is to run through all possible integer values and then total and average the normalized results for each approach:

using System;

class Program
{
   static void Main()
   {
       int min = 1;
       int max = 100; 
       Normalization norm = new Normalization(min, max);
       double total = 0.0;
       double total2 = 0.0;

       for(int i = min; i <= max; i++)
       {
          total  += norm.LogNormalize(i);
          total2 += norm.LogNormalize2(i);        
       }      

       Console.WriteLine("Average value when adding one is :     {0:F3}", total/100.00);
       Console.WriteLine("Average value when not adding one is : {0:F3}", total2/100.00);

       Console.ReadLine();
   }

}

public class Normalization
{
    public readonly int Min, Max, Range;
    public readonly double Lower, Higher, Interval;

    public Normalization(int min,int max, double lower, double higher)
    {
       Min = min;
       Max = max;
       Range = max - min + 1;     
       Lower = lower;
       Higher = higher;
       Interval = higher - lower;
    }

    public Normalization(int min, int max): this(min, max, 0, 1)
    {
    }

    public double Normalize(int i)
    {
       if (i <= Min) return Lower;
       if (i >= Max) return Higher;
       return Lower + (double)(i - Min)/(Range - 1) * Interval;
    }

    // add one to the logs
    public double LogNormalize(int i)
    {
       if (i <= Min) return Lower;
       if (i >= Max) return Higher;
       double temp = Math.Log(i - Min + 1)/Math.Log(Max - Min + 1);
       return Lower + temp * Interval;
    }

    // don't add one to the logs
    public double LogNormalize2(int i)
    {
       if (i <= Min) return Lower;
       if (i >= Max) return Higher;
       double temp = Math.Log(i - Min)/Math.Log(Max - Min);
       return Lower + temp * Interval;
    }

}

The results were 0.790 when adding one and 0.782 when not, so I think the answer is that it doesn't make much difference. However, if your actual range of numbers is less than 100, then you may want to run this code using the actual range to make sure the difference is still acceptable. 

Thanks Alan,

 

I want to analyze whether adding 1 will impact the result of logarithm based normalization algorithm, but do not have enough knowledge. Do you have any ideas? Does it impact or not or not so much?

 

 

regards,

George

Hi George,

The reason why I defined Range as Max - Min + 1 is because that's what the range actually is when we include both endpoints and because I was exposing it as a public readonly field where the caller might expect such a value to be returned.

In the logarithmic code, I added 1 to the numerator to prevent  Math.Log(i - Min) becoming 0 when i was equal to Min + 1 which would have caused the method to return a value of 0 for both i == Min and i == Min +1. Having added it to the numerator, I then had to add it to the denominator as well to prevent a value of 1 being returned for both i == Max - 1 and i == Max.

Although it seems a bit crude, I believe it's a standard technique when dealing with logarithmic interpolation.

Cool, Alan!

 

 

Two comments,

 

1. In your original linear code, why not change to the following code to save time to minus 1 each time?

 

Range = max - min;

return Lower + (double)(i - Min)/(Range ) * Interval;

 

2.

 

Why in your logarithm code, you add 1 to both Math.Log(i - Min + 1) and Math.Log(Max - Min + 1)?

 

 

regards,

George

Firstly, I've just noticed that there's a mistake in my normalization formula (even though it works correctly when the interval starts at 0) in that 'Lower' needs to be added on to the result. The corrected method is therefore:

public double Normalize(int i)
{
       if (i <= Min) return Lower;
       if (i >= Max) return Higher;
       return Lower + (double)(i - Min)/(Range - 1) * Interval;
}

I'd imagine that with log based normalization, you use logarithmic rather than linear interpolation to determine where the integer value will map within the interval. That would suggest something like:

public double LogNormalize(int i)
{
       if (i <= Min) return Lower;
       if (i >= Max) return Higher;
       double temp = Math.Log(i - Min + 1)/Math.Log(Max - Min + 1);
       return Lower + temp * Interval;
}

However, this isn't going to be suitable for all distributions as it skews the normalized values to the upper half of the interval. In the example program, a value of 0.5 corresponds to an integer of 10 and so 90% of the range will be above 0.5.

Cool,

 

I heard there is also a logarithm based normalization approach, do you know how to implement in this way?

 

 

regards,

George

Something like this, perhaps:

using System;

class Program
{
   static void Main()
   {
       Normalization norm = new Normalization(1, 100);
       double result = norm.Normalize(50);
       Console.WriteLine(result);
       Console.ReadLine();
   }

}

public class Normalization
{
    public readonly int Min, Max, Range;
    public readonly double Lower, Higher, Interval;

    public Normalization(int min,int max, double lower, double higher)
    {
       Min = min;
       Max = max;
       Range = max - min + 1;     
       Lower = lower;
       Higher = higher;
       Interval = higher - lower;
    }

    public Normalization(int min, int max): this(min, max, 0, 1)
    {
    }

    public double Normalize(int i)
    {
       if (i <= Min) return Lower;
       if (i >= Max) return Higher;
       return (double)(i - Min)/(Range - 1) * Interval;
    }
}