Any interests in element wise operations on vector/matrix with F#?


(Albert Pang) #1

Hi all, would anyone be interested in having matrix/vector support (or expose) element wise mathematical operations when using them in F#? Something like the below:

> open MathNet.Numerics.LinearAlgebra;;
> let m = matrix [[1.0; 2.0; 3.0]; [0.5; 1.1; 2.1]];;

val m : Matrix = DenseMatrix 2x3-Double
1 2 3
0.5 1.1 2.1

log m;;
val it : Matrix =
DenseMatrix 2x3-Double
0 0.693147 1.09861
-0.693147 0.0953102 0.741937

{ColumnCount = 3;
 Item = ?;
 RowCount = 2;
 Storage = MathNet.Numerics.LinearAlgebra.Storage.DenseColumnMajorMatrixStorage`1[System.Double];
 Values = [|0.0; -0.6931471806; 0.6931471806; 0.0953101798; 1.098612289;
            0.7419373447|];}

exp m;;
val it : Matrix =
DenseMatrix 2x3-Double
2.71828 7.38906 20.0855
1.64872 3.00417 8.16617

{ColumnCount = 3;
 Item = ?;
 RowCount = 2;
 Storage = MathNet.Numerics.LinearAlgebra.Storage.DenseColumnMajorMatrixStorage`1[System.Double];
 Values = [|2.718281828; 1.648721271; 7.389056099; 3.004166024;
            20.08553692; 8.166169913|];}

m .^ 2.0;;
val it : Matrix =
DenseMatrix 2x3-Double
1 4 9
0.25 1.21 4.41

{ColumnCount = 3;
 Item = ?;
 RowCount = 2;
 Storage = MathNet.Numerics.LinearAlgebra.Storage.DenseColumnMajorMatrixStorage`1[System.Double];
 Values = [|1.0; 0.25; 4.0; 1.21; 9.0; 4.41|];}

The last one is mimicking the octave element wise exponent operator.

All it takes to have these in F# is for the point-wise operations to be exposed as a static member function with a specific name, e.g. Log() for ‘log’, Exp() for ‘exp’ in the Matrix and Vector types. Implementation wise, PointwiseLog() and PointwiseExp() are already there but they are not yet exposed (please correct me if I am wrong). I have added some code to implement all the operators under “Math Operators” in this post: https://blogs.msdn.microsoft.com/dsyme/2008/09/01/the-f-operators-and-basic-functions/ plus exposing the existing PointwiseLog() and PointwiseExp() functions in Matrix.Operators.cs and Vector.Operators.cs.

If any one is interested in having these, I would gladly send a pull request. Or if this is actually not a very good idea for whatever reasons, please kindly share.

Thanks for reading!

Cheers,
Albert


(Tyetis) #2

Hi Albert,

I’m not interested matrix or vector but i have a project about equation solving and linear algebra. do you are interested in this ?


(Christoph Rüegg) #3

We already support a few of the F# pointwise operators, specifically .*, ./ and .%. Pointwise power is indeed missing and should be added. Thanks for the reminder! A pull request would of course be welcome :).

Adding support for more functions like exp and log would be interesting, but we need to be careful. There is a reason the pointwise exponential is currently provided as PointwiseExp and not Exp: the matrix exponential is actually defined for square matrices (even though we do not implement it), and very different from the pointwise exponential.


(Christoph Rüegg) #4

Also, I wonder whether we can do anything to suppress the extra output in FSI, let me know if you have any idea.


(Albert Pang) #5

Ah, I see. That’s the reason. I am guessing you are referring to these definitions of exp and log for a square matrix: https://www.gnu.org/software/octave/doc/v4.0.1/Functions-of-a-Matrix.html#Functions-of-a-Matrix

How about we add pointwise operations for these functions below:
abs, acos, asin, atan, ceiling, cos, cosh, floor, log10, round, sign, sin, sinh, sqrt, tan, tanh

We will simply name these functions along the same line as PointwiseExp(). e.g. PointwiseAbs(), PointwiseAcos(), etc. These should then do no harm as it is obvious that we are referring to a pointwise operation even if there exists other definitions for some of these operations.

Now as to exposing them as static members, we will avoid exp and log. So that it will not be valid any more to do the below

> let a = matrix [[1.0; 2.0; 3.0]; [4.0; 5.0; 6.0]];;

val a : Matrix = DenseMatrix 2x3-Double
1 2 3
4 5 6

exp a;;

exp a;;
----^

C:\Users\panga\AppData\Local\Temp\stdin(15,5): error FS0001: The type ‘Matrix’ does not support the operator ‘Exp’

But we do expose all the others except exp and log. so that for example, this will work for abs and I can imagine it will be quite useful at some point:

X;;
val it : Matrix =
DenseMatrix 47x3-Double
1 0.13001 -0.223675
1 -0.50419 -0.223675
1 0.502476 -0.223675
1 -0.735723 -1.53777
1 1.25748 1.09042
1 -0.0197317 1.09042
1 -0.58724 -0.223675
1 -0.721881 -0.223675
… … …
1 -1.00752 -0.223675
1 -1.44542 -1.53777
1 -0.18709 1.09042
1 -1.00375 -0.223675

abs X;;
val it : Matrix =
DenseMatrix 47x3-Double
1 0.13001 0.223675
1 0.50419 0.223675
1 0.502476 0.223675
1 0.735723 1.53777
1 1.25748 1.09042
1 0.0197317 1.09042
1 0.58724 0.223675
1 0.721881 0.223675
… … …
1 1.00752 0.223675
1 1.44542 1.53777
1 0.18709 1.09042
1 1.00375 0.223675

Also, we create op_DotHat to denote PointwisePower() (this is mimicking Octave) So we can do:

> X .^ 2.0;;
val it : Matrix =
DenseMatrix 47x3-Double
1 0.0169026 0.0500306
1 0.254207 0.0500306
1 0.252482 0.0500306
1 0.541288 2.36473
1 1.58125 1.18901
1 0.000389341 1.18901
1 0.344851 0.0500306
1 0.521113 0.0500306
… … …
1 1.0151 0.0500306
1 2.08925 2.36473
1 0.0350027 1.18901
1 1.00751 0.0500306

In addition, I have added a couple of functions to extend the Matrix and Vector module in F#. They are very short, so I might as well just show them here:

module Vector =
let inline ofSeq s = s |> Seq.toList |> vector

module Matrix =
let inline aggCols f m = m |> Matrix.toColSeq |> Seq.map f |> Vector.ofSeq
let inline replicateRows n v = v|> Vector.toList |> List.replicate n |> matrix
let inline replicateCols n v = replicateRows n v |> Matrix.transpose

All these will let me implement gradient descent for linear regression and logistic regression quite nicely. See below:

Feature normalization can be done like this:

let featureNormalize (x: Matrix) =
let repl v = Matrix.replicateRows x.RowCount v
let mu = x |> Matrix.aggCols Statistics.Mean |> repl
let sigma = x |> Matrix.aggCols Statistics.StandardDeviation |> repl
mu, sigma, (x - mu) ./ sigma

And the theta update function can be done like this:

let inline lgrTheta (alpha: float) (x: Matrix) (y: Vector) t =
let m = float x.RowCount
let sigmoid (x:Vector) = 1.0 / (1.0 + (-x).PointwiseExp())
t - alpha / m * (x.Transpose()) * (sigmoid(x*t) - y)

If we have exposed PointwiseExp, our sigmoid function above can be written even more nicely:

let sigmoid (x: Vector) = 1.0 / (1.0 + exp (-x))

But then writing PointwiseExp() isn’t too bad either.

What do you (and everyone else) think? I have done the above in a fork of mine. Please feel free to check it out at https://github.com/albertp007/mathnet-numerics

Cheers,
Albert


(Christoph Rüegg) #6

To be precise, all of the trigonometric functions are defined on matrices as well, along the same lines as exp and log (see Trigonometric functions of matrices). We implement none of them of course, but if we ever would, we might run into a problem.

On the other hand, I expect the point wise operations to be used much more often in practice. Usability is important as well. Thanks for putting these together!


(Christoph Rüegg) #7

Just to be clear, I’d really like to have great usability and readability in F#. exp X is much more readable than X.PointwiseExp().


(Albert Pang) #8

Got it - basically any function that has a definition through taylor series expansion. Thanks for the share!

I was about to send a pull request :slight_smile:

Let me pull exp and log back in then. I took out “sign” from being a static member though because I just realized it won’t work in F# which requires it to return an integer and therefore is completely different from a pointwise operation.

I have other suggestions on the F# modules, but I think I would start a different thread as these are just minor enhancements and are not really related to the pointwise operations we discussed in this thread…

Cheers,
Albert


(Christoph Rüegg) #9

Great, thanks!

It might be an option to offer matrix functions which are essentially extended to matrices through taylor expansion with a matrix prefix, if we ever add them in the future, e.g. MatrixExp, MatrixSin. Then we could reserve the form without prefix for the more common use case.


(Christoph Rüegg) #10

For reference: GitHub Pull Request