[PyCUDA] petsc4py+PyCUDA through Cython?

[Ashwin Srinath]

Hello, PyCUDA users!

I'm trying to construct a GPUArray from device memory allocated using
petsc4py. I've written some C code that extracts a raw pointer from a PETSc
"cusp" vector. Now, I am hoping to 'place' this memory into a gpuarray,
using PyCUDA and Cython in some way. Here's a Cython snippet that I hope
will make it clear:

def getGPUArray(Vec V):
cdef double *array
VecGetGPUArray(V.vec, &array)
G = gpuarray.??? # how to construct this gpuarray?
print G.get()

`VecGetGPUArray` is some C code that extracts the raw pointer from Vec.
`array` is the raw pointer from which I want to be able to construct a
gpuarray.

I think this could be very useful to petsc4py users, and I'd be happy to
provide more information/code to make this work :)

Thanks so much,
Ashwin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141013/c1e65423/attachment.html>

[Andreas Kloeckner]

Post by Ashwin Srinath
Hello, PyCUDA users!
I'm trying to construct a GPUArray from device memory allocated using
petsc4py. I've written some C code that extracts a raw pointer from a PETSc
"cusp" vector. Now, I am hoping to 'place' this memory into a gpuarray,
using PyCUDA and Cython in some way. Here's a Cython snippet that I hope
cdef double *array
VecGetGPUArray(V.vec, &array)
G = gpuarray.??? # how to construct this gpuarray?
print G.get()
`VecGetGPUArray` is some C code that extracts the raw pointer from Vec.
`array` is the raw pointer from which I want to be able to construct a
gpuarray.
I think this could be very useful to petsc4py users, and I'd be happy to
provide more information/code to make this work :)

The GPUArray constructor has an undocumented data= keyword argument that
you should be able to use.

Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 810 bytes
Desc: not available
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141014/379c87b6/attachment.sig>

[Ashwin Srinath]

Thank you for your reply, Andreas, and for your work!

Keeping in mind that `array` is of type double*, would this be the right
way to do it? I realize this is more a Cython question:

cdef extern from "stdint.h":
ctypedef unsigned long long uint64_t

V_gpu = gpuarray.empty(V.getSizes(), dtype=np.float64,
gpudata=<uint64_t>array)

Thank you
Ashwin

On Tue, Oct 14, 2014 at 2:41 AM, Andreas Kloeckner <lists at informa.tiker.net>

Post by Andreas Kloeckner

Post by Ashwin Srinath
Hello, PyCUDA users!
I'm trying to construct a GPUArray from device memory allocated using
petsc4py. I've written some C code that extracts a raw pointer from a

PETSc

Post by Ashwin Srinath
"cusp" vector. Now, I am hoping to 'place' this memory into a gpuarray,
using PyCUDA and Cython in some way. Here's a Cython snippet that I hope
cdef double *array
VecGetGPUArray(V.vec, &array)
G = gpuarray.??? # how to construct this gpuarray?
print G.get()
`VecGetGPUArray` is some C code that extracts the raw pointer from Vec.
`array` is the raw pointer from which I want to be able to construct a
gpuarray.
I think this could be very useful to petsc4py users, and I'd be happy to
provide more information/code to make this work :)

The GPUArray constructor has an undocumented data= keyword argument that
you should be able to use.
Andreas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141014/5d51565c/attachment.html>

[Andreas Kloeckner]

Post by Ashwin Srinath
Thank you for your reply, Andreas, and for your work!
Keeping in mind that `array` is of type double*, would this be the right
ctypedef unsigned long long uint64_t
V_gpu = gpuarray.empty(V.getSizes(), dtype=np.float64,
gpudata=<uint64_t>array)

Yes, I think so. Perhaps cast gpudata to a bare int if there's
trouble. Also there are concerns of who owns the data and how long it
lives, but if you don't need that, then you should be set.

Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 810 bytes
Desc: not available
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141014/c678ab82/attachment.sig>

[Ashwin Srinath]

I'd like to report partial success! Following your hint worked, and now I'm
successfully able to construct a GPUArray from a PETSc vector in Python!

However, as you pointed out, I should be concerned about who owns what.
When I make changes to the GPUArray, they are not reflected in the CUSP
vector! You can view my test script here:

https://github.com/ashwinsrnth/petsc-pycuda/blob/master/run_demo.py

The output I get:

Original PETSc Vec:
[ 2. 2. 2. 2.]
GPUArray constructed from PETSc Vec:
[ 2. 2. 2. 2.]
Modified GPUArray:
[ 4. 4. 4. 4.]
PETSc Vec changes too:
[ 2. 2. 2. 2.]

Ideally, that last line should be [4., 4., 4., 4.].

Any hints to make this work?

Thank you!
Ashwin


On Tue, Oct 14, 2014 at 10:10 AM, Andreas Kloeckner <lists at informa.tiker.net

Post by Andreas Kloeckner

Post by Ashwin Srinath
Thank you for your reply, Andreas, and for your work!
Keeping in mind that `array` is of type double*, would this be the right
ctypedef unsigned long long uint64_t
V_gpu = gpuarray.empty(V.getSizes(), dtype=np.float64,
gpudata=<uint64_t>array)

Yes, I think so. Perhaps cast gpudata to a bare int if there's
trouble. Also there are concerns of who owns the data and how long it
lives, but if you don't need that, then you should be set.
Andreas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141014/f31b63d1/attachment.html>

[Ashwin Srinath]

I'm not sure - but this may have something to do with the implementation of
`fill`. Because on the flip side, changes to the PETSc Vec *are* reflected
the GPUArray. So I can see that they are actually sharing device memory..

Thanks,
Ashwin

On Tue, Oct 14, 2014 at 7:55 PM, Ashwin Srinath <ashwinsrnth at gmail.com>

Post by Ashwin Srinath
I'd like to report partial success! Following your hint worked, and now
I'm successfully able to construct a GPUArray from a PETSc vector in Python!
However, as you pointed out, I should be concerned about who owns what.
When I make changes to the GPUArray, they are not reflected in the CUSP
https://github.com/ashwinsrnth/petsc-pycuda/blob/master/run_demo.py
[ 2. 2. 2. 2.]
[ 2. 2. 2. 2.]
[ 4. 4. 4. 4.]
[ 2. 2. 2. 2.]
Ideally, that last line should be [4., 4., 4., 4.].
Any hints to make this work?
Thank you!
Ashwin
On Tue, Oct 14, 2014 at 10:10 AM, Andreas Kloeckner <

Post by Andreas Kloeckner

Post by Ashwin Srinath
Thank you for your reply, Andreas, and for your work!
Keeping in mind that `array` is of type double*, would this be the right
ctypedef unsigned long long uint64_t
V_gpu = gpuarray.empty(V.getSizes(), dtype=np.float64,
gpudata=<uint64_t>array)

Yes, I think so. Perhaps cast gpudata to a bare int if there's
trouble. Also there are concerns of who owns the data and how long it
lives, but if you don't need that, then you should be set.
Andreas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141014/a47a06fc/attachment.html>

[Andreas Kloeckner]

Post by Ashwin Srinath
I'm not sure - but this may have something to do with the implementation of
`fill`. Because on the flip side, changes to the PETSc Vec *are* reflected
the GPUArray. So I can see that they are actually sharing device memory..

As far as I know, PETSc maintains a copy of the vector on the host and
on the device and tacitly assumes that nobody except it makes
modifications to the vector. So if it assumes nobody has modified the
device-side vector, it may give you a (stale) host-side copy.

HTH,
Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 810 bytes
Desc: not available
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141016/220f4781/attachment.sig>

[Ashwin Srinath]

Thank you, Andreas. The documentation does mention that PETSc internally
keeps track of which (host or device) vector is last updated. So when I
update the memory on the PyCUDA side, maybe PETSc doesn't know about it.

Thank you, I'll investigate further.

On Thu, Oct 16, 2014 at 1:28 PM, Andreas Kloeckner <lists at informa.tiker.net>

Post by Ashwin Srinath

Post by Ashwin Srinath
I'm not sure - but this may have something to do with the implementation

of

Post by Ashwin Srinath
`fill`. Because on the flip side, changes to the PETSc Vec *are*

reflected

Post by Ashwin Srinath
the GPUArray. So I can see that they are actually sharing device memory..

As far as I know, PETSc maintains a copy of the vector on the host and
on the device and tacitly assumes that nobody except it makes
modifications to the vector. So if it assumes nobody has modified the
device-side vector, it may give you a (stale) host-side copy.
HTH,
Andreas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141016/82a8a743/attachment.html>

[Ashwin Srinath]

Yup. That was it. Manually updating this flag after PyCUDA modified the
buffer fixed it. Thank you!

And if anyone is interested, here is a petsc4py extension that lets you
access PETSc vectors as PyCUDA GPUArrays:
https://github.com/ashwinsrnth/petsc-pycuda

On Thu, Oct 16, 2014 at 1:37 PM, Ashwin Srinath <ashwinsrnth at gmail.com>

Post by Ashwin Srinath
Thank you, Andreas. The documentation does mention that PETSc internally
keeps track of which (host or device) vector is last updated. So when I
update the memory on the PyCUDA side, maybe PETSc doesn't know about it.
Thank you, I'll investigate further.
On Thu, Oct 16, 2014 at 1:28 PM, Andreas Kloeckner <

Post by Ashwin Srinath

Post by Ashwin Srinath
I'm not sure - but this may have something to do with the

implementation of

Post by Ashwin Srinath
`fill`. Because on the flip side, changes to the PETSc Vec *are*

reflected

Post by Ashwin Srinath
the GPUArray. So I can see that they are actually sharing device

memory..
As far as I know, PETSc maintains a copy of the vector on the host and
on the device and tacitly assumes that nobody except it makes
modifications to the vector. So if it assumes nobody has modified the
device-side vector, it may give you a (stale) host-side copy.
HTH,
Andreas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.tiker.net/pipermail/pycuda/attachments/20141016/e44bc125/attachment.html>