Clarify benchmarks?

[agibsonccc]

Hey folks:

First of all, I'm kind of dismayed you guys didn't talk to us about your findings. You guys are making some heavy claims. We are about to do a release this week. We were busy essentially writing our own TF including import.

From an initial look, you guys didn't do your benchmarks properly.

1. You are missing workspaces in your microbenchmarks which defeats the purpose of
   benchmarking nd4j.
2. You don't show any of nd4j's native configuration or any of the memory configurations you guys tried.
   I know it's in your guys' interest to have your own framework. That's up to you.

We'll do a blog post to correct a lot of these misconceptions you guys are perpetuating here, but in the mean time, we'll do our best to clarify questions you guys have. We don't mind having competition. It's great to keep us on our toes, but we want to make sure anything representing us is at least somewhat fair (even just handling some of the lower hanging fruit)

[raver119]

Unfortunately i'm not Scala guy, so i'd like to ask few more questions regarding tests equality, if you don't mind.

https://github.com/ThoughtWorksInc/Compute.scala/blob/nvidia-gpu/benchmarks/src/jmh/scala/com/thoughtworks/compute/benchmarks.scala#L144-L181

Line 144: Was number of threads set to 1 in runtime? Or number of CPU threads were > 1 during both tests?
Lines 153 & 168: Does that means you've included input generation time to Tanh measurements?
Line 170: So, you do use "cache" for your library, but not using workspaces for ND4j? Nice.
Line 174: foldLeft InlineTensor does in this case it means that operation in this case is executed in-place? As in "input array gets modified and returned back" once .flatArray() method is called?

[Atry]

2018-04-02 13:21 GMT+08:00 raver119 <notifications@github.com>:

Unfortunately i'm not Scala guy, so i'd like to ask few more questions regarding tests equality, if you don't mind. https://github.com/ThoughtWorksInc/Compute.scala/ blob/nvidia-gpu/benchmarks/src/jmh/scala/com/thoughtworks/compute/ benchmarks.scala#L144-L181 Line 144: Was number of threads set to 1 in runtime? Or number of CPU threads were > 1 during both tests?

CPU threads were > 1 during both tests. I suggest you learn to use JMH, because it is a very good tool when you are implementing a performance-critical framework.

Lines 153 & 168: Does that means you've included input generation time to Tanh measurements?

The input generation will be completed in warm-up iterations. I recommend you read this: http://fdahms.com/2015/10/14/scala-and-the-transient-lazy-val-pattern/ . I hope the article may help you better understand how Scala lazy val works.

Line 170: So, you do use "cache" for your library, but not using workspaces for ND4j? Nice.

The cache method is an equivalent to lazy val in the ND4J initialization. It just allocate a buffer for input, not for the rest computation.

Line 174: foldLeft InlineTensor does in this case it means that operation in this case is executed in-place? As in "input array gets modified and returned back" once .flatArray() method is called?

Compute.scala does not support any in-place mutable operation. The foldLeft InlineTensor means merging multiple tanh calls into one kernel program.

…

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#137 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAktupy5LaIGFoPJTNtpOa4Q5DBCMKcyks5tkbV3gaJpZM4TDI70> .

-- 杨博 (Yang Bo)

[raver119]

On Apr 1, 2018, at 23:18, 杨博 (Yang Bo) ***@***.***> wrote: 2018-04-02 13:21 GMT+08:00 raver119 ***@***.***>: > Unfortunately i'm not Scala guy, so i'd like to ask few more questions > regarding tests equality, if you don't mind. > > https://github.com/ThoughtWorksInc/Compute.scala/ > blob/nvidia-gpu/benchmarks/src/jmh/scala/com/thoughtworks/compute/ > benchmarks.scala#L144-L181 > > Line 144: Was number of threads set to 1 in runtime? Or number of CPU > threads were > 1 during both tests? > CPU threads were > 1 during both tests. I suggest you learn to use JMH, because it is a very good tool when you are implementing a

I’m sorry, bu my question actually had a bit deeper sense. JMH allows runtime override for Thread annotation, and the only thing i’ve asked you - if there was override when you was running your code or no. I take your answer as no, though. So next question is: what’s the idea of testing CUDA in multiple parallel CPU threads? Was workload small enough? p.s. Thanks for advice.

> Lines 153 & 168: Does that means you've included input generation time to > Tanh measurements? > The input generation will be completed in warm-up iteration. I recommend you read this: http://fdahms.com/2015/10/14/scala-and-the-transient-lazy- val-pattern/ . I hope the article may help you better understand how Scala lazy val works.

Oh, interesting.

> Line 170: So, you do use "cache" for your library, but not using > workspaces for ND4j? Nice. > The cache method is an equivalent to lazy val in the ND4J initialization. It just allocate a buffer for input, not for the rest computation.

Perfect. See next question below please.

> Line 174: foldLeft InlineTensor does in this case it means that operation > in this case is executed in-place? As in "input array gets modified and > returned back" once .flatArray() method is called? > Compute.scala does not support any in-place mutable operation. The foldLeft InlineTensor means merging multiple tanh call into one kernel program.

Ok. Let me ask in other words then. Lines 170:178. There’s loop with `numberOfIterations` iterations. How many independent memory buffers allocated there? 0? 1? 2? `numberOfIterations` or `numberOfIterations` x 2? EDIT: I mean off-heap buffers, available for the GPU.

…

> — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#137 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAktupy5LaIGFoPJTNtpOa4Q5DBCMKcyks5tkbV3gaJpZM4TDI70> > . > -- 杨博 (Yang Bo) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#137 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALru_3AdMzdzvQ-F0oKeMDAEKTcKPmv5ks5tkcLQgaJpZM4TDI70>.

[Atry]

2018-04-02 14:49 GMT+08:00 raver119 <notifications@github.com>:

> On Apr 1, 2018, at 23:18, 杨博 (Yang Bo) ***@***.***> wrote: > > 2018-04-02 13:21 GMT+08:00 raver119 ***@***.***>: > > > Unfortunately i'm not Scala guy, so i'd like to ask few more questions > > regarding tests equality, if you don't mind. > > > > https://github.com/ThoughtWorksInc/Compute.scala/ > > blob/nvidia-gpu/benchmarks/src/jmh/scala/com/thoughtworks/compute/ > > benchmarks.scala#L144-L181 > > > > Line 144: Was number of threads set to 1 in runtime? Or number of CPU > > threads were > 1 during both tests? > > > CPU threads were > 1 during both tests. I suggest you learn to use JMH, > because it is a very good tool when you are implementing a I’m sorry, bu my question actually had a bit deeper sense. JMH allows runtime override for Thread annotation, and the only thing i’ve asked you - if there was override when you was running your code or no. I take your answer as no, though. So next question is: what’s the idea of testing CUDA in multiple parallel CPU threads? Was workload small enough? p.s. Thanks for advice. > > > Lines 153 & 168: Does that means you've included input generation time to > > Tanh measurements? > > > The input generation will be completed in warm-up iteration. I recommend > you read this: http://fdahms.com/2015/10/14/ scala-and-the-transient-lazy- > val-pattern/ . I hope the article may help you better understand how Scala > lazy val works. Oh, interesting. > > > Line 170: So, you do use "cache" for your library, but not using > > workspaces for ND4j? Nice. > > > The cache method is an equivalent to lazy val in the ND4J initialization. > It just allocate a buffer for input, not for the rest computation. Perfect. See next question below please. > > Line 174: foldLeft InlineTensor does in this case it means that operation > > in this case is executed in-place? As in "input array gets modified and > > returned back" once .flatArray() method is called? > > > Compute.scala does not support any in-place mutable operation. The > foldLeft InlineTensor means merging multiple tanh call into one kernel > program. Ok. Let me ask in other words then. Lines 170:178. There’s loop with `numberOfIterations` iterations. How many independent memory buffers allocated there? 0? 1? 2? `numberOfIterations` or `numberOfIterations` x 2?

1 buffer, which stores the result. I know it's not fair to ND4J, but I really don't know how to make ND4J merge multiple immutable operations into one.

…

> > — > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub > > <https://github.com/ThoughtWorksInc/Compute.scala/ issues/137#issuecomment-377857399>, > > or mute the thread > > <https://github.com/notifications/unsubscribe-auth/ AAktupy5LaIGFoPJTNtpOa4Q5DBCMKcyks5tkbV3gaJpZM4TDI70> > > . > > > > > > -- > 杨博 (Yang Bo) > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub <https://github.com/ ThoughtWorksInc/Compute.scala#137#issuecomment-377864516>, or mute the thread <https://github.com/notifications/unsubscribe- auth/ALru_3AdMzdzvQ-F0oKeMDAEKTcKPmv5ks5tkcLQgaJpZM4TDI70>. > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#137 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAktuoIdnTyUL1bad4o3paoE0wh4YH4jks5tkcnzgaJpZM4TDI70> .

-- 杨博 (Yang Bo)

[Atry]

You are very keen to realize that the performance issue in the benchmark is related to the memory. For memory settings, you can check the benchmark source code. JMH forks JVM when running, the JVM flags are provided by annotations. No annotation means default JVM configuration. For workspaces, I have to confess that I am not familiar with workspace. But the purpose of this Compute.scala library is allowing people do not care about memory even when they are using arbitrary immutable operations. I understand a carefully optimized application written in ND4J is good. But it seems that many users of ND4J are not smart enough to avoid OutOfMemoryError: https://github.com/deeplearning4j/deeplearning4j/issues?utf8=%E2%9C%93&q=OutOfMemoryError 2018-04-02 12:35 GMT+08:00 Adam Gibson <notifications@github.com>:

…

Hey folks: First of all, I'm kind of dismayed you guys didn't talk to us about your findings. You guys are making some heavy claims. We are about to do a release this week. We were busy essentially writing our own TF including import. From an initial look, you guys didn't do your benchmarks properly. 1. You are missing workspaces in your microbenchmarks which defeats the purpose of benchmarking nd4j. 2. You don't show any of nd4j's native configuration or any of the memory configurations you guys tried. I know it's in your guys' interest to have your own framework. That's up to you. We'll do a blog post to correct a lot of these misconceptions you guys are perpetuating here, but in the mean time, we'll do our best to clarify questions you guys have. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#137>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAktusVBZIKmpA_yOJepxg5QPMjFuVssks5tkaqZgaJpZM4TDI70> .

-- 杨博 (Yang Bo)

[Atry]

2018-04-02 14:49 GMT+08:00 raver119 <notifications@github.com>:

I’m sorry, bu my question actually had a bit deeper sense. JMH allows runtime override for Thread annotation, and the only thing i’ve asked you - if there was override when you was running your code or no. I take your answer as no, though. So next question is: what’s the idea of testing CUDA in multiple parallel CPU threads? Was workload small enough? p.s. Thanks for advice.

Yes, the purpose is to avoid GPU starving. I did not test but I guess Compute.scala will even be faster on larger arrays with less threads, because it reduces the overhead of the driver, considering NVIDIA OpenCL driver has a higher overhead than CUDA.

…

-- 杨博 (Yang Bo)

[raver119]

I see.

This benchmark isn't comparing apples to apples. Thanks for your time.

[Atry]

You can see there is a notice for Deeplearning4j in the README.md.

We have never criticized the performance of ND4J's in a mutable style.

[Atry]

2018-04-02 22:27 GMT+08:00 raver119 <notifications@github.com>:

I see. This benchmark isn't comparing apples to apples. Thanks for your time.

You are wrong. The benchmark is comparing apples to apples: immutable operations vs immutable operations Unfortunately, ND4J does not have the feature of dynamic kernel generation, or JIT in PyTorch's terminology. For example, when you compare the performance of Lua runtimes between Lua and Luajit . You can say it's not fair because Luajit support JIT. But it's still comparing the same semantics of operations. I understand ND4J is designed for Deeplearning4j, which only needs mutable in-place operations, so you guys don't have to optimize the performance for other usage.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <#137 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAktuqoiJatr2nUyAnzR4iHj280WRVmuks5tkjU-gaJpZM4TDI70> .

-- 杨博 (Yang Bo)

[raver119]

You are wrong. The benchmark is comparing apples to apples: immutable
operations vs immutable operations

Few messages above you've said that 1 array was allocated for the loop. Now you say: it's immutable vs immutable comparison. So which answer is correct?

I.e. your code that uses Nd4j does numberOfIterations x2 allocations, because your Transform.tanh() call creates new INDArray each time, and each INDArray has 2 buffers - 1 on gpu side, 1 on host side. With 5 total iterations your test basically benchmarks CUDA allocation performance, and not actual tanh.

If you call that "apples to apples comparison" - okay, that's up to you :)

re Nd4j for Dl4j. Nd4j just mimics numpy basically.

[Atry]

2018-04-03 0:41 GMT+08:00 raver119 <notifications@github.com>:

You are wrong. The benchmark is comparing apples to apples: immutable operations vs immutable operations Few messages above you've said that 1 array was allocated for the loop. Now you say: it's immutable vs immutable comparison. So which answer is correct? I.e. your code that uses Nd4j does numberOfIterations x2 allocations, because your Transform.tanh() call creates new INDArray each time, and each INDArray has 2 buffers - 1 on gpu side, 1 on host side. With 5 total iterations your test basically benchmarks CUDA allocation performance, and not actual tanh.

You are talking about your implementation, not the behavior. You provide a very good explanation why ND4J's implementation consumes more memory than Compute.scala for the same behavior, which is exactly what the benchmark demonstrated.

…

If you call that "apples to apples comparison" - okay, that's up to you :) re Nd4j for Dl4j. Nd4j just mimics numpy basically. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#137 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAktut0T_xNV80V8BYAugf0aYGD_FqS_ks5tklSngaJpZM4TDI70> .

-- 杨博 (Yang Bo)

[raver119]

No, i've just explained that you hadn't understood how to implement stuff with ND4j efficiently, and did claims like this:

ND4J's implementation consumes more memory

It's not about ND4j implementation. It's about what YOU've implemented. Because obviously the same code could be written without allocating new INDArray on each iteration. Just difference of 1 argument :)

[raver119]

P.d. don't get me wrong please. Personally i don't care about your claims etc. You want to claim you're faster then Nd4j? I'm ok with that. If you'll want to claim that you're faster then light - I'll be ok with that as well.

The only reason I was here - is performance feedback. When i hear about Nd4j performance problems - i'm always trying to get to the bottom of the problem, and improve whatever is possible to improve. In this particular case i see - it's a waste of time for me, due to various reasons. Different approaches, bad benchmarking setup, different goals etc.

Thanks for your time.

[Atry]

2018-04-03 0:49 GMT+08:00 raver119 <notifications@github.com>:

It's not about ND4j implementation. It's about what YOU've implemented.

Suppose a data scientist Alice read a paper and want to reproduce an algorithm in the paper, say, `a * b + c`. She found that the ND4J/ND4S version of `a * b + c` consumes four times memory than the Compute.scala version of `a * b + c`. And Skymind guys blame Alice because she did not refactor her ND4J version to `a *= b; a += c`. Interesting...

…

-- 杨博 (Yang Bo)

[raver119]

Imagine Alice does some reading of documentation, and instead of:

a.mul(b).addi(c)

does something like:

a.muli(b).addi(c)

Thats when it becomes interesting... :)